PDA

View Full Version : Live time black dots


birdboy
6th November 2013, 12:27 PM
Does anyone use live time and what sort of results do you get.

I have been getting black dots akin to dead pixels in the shadows with my OM1. Now I would assume that live time is mainly used in very low light conditions, fireworks light trials night shoots etc. I can repeat the problem at will. The black dots are not random but specific pixels. You will need to pixel peep to see these black dots but they do seriously degrade the IQ of the picture so it should be an issue.

It seems that at least one other OM5 user has experienced the black dots, so is this an issue with the new sensor and the live time feature.

I have an issue raised with Olympus technical support but am getting nowhere, they keep putting it down to noise artefacts which it is not because the black dots are in specific pixel locations.

Anyone with examples of live time shots?
Thanks John

ps Ian is their any chance that you could you your influence with the powers that be to look into this. My experience is that the problem is easily to repeat so their should be no problem confirming if this is an issue or not.

photo_owl
6th November 2013, 12:50 PM
I'll see what I can do - but from my understanding of your previous post what we are really looking at is "what's the camera doing differently when shooting LT and bulb?"

then whatever that difference is must, in some way, be delivering those black dots.

are we on the same wavelength?

btw why should they be specific to the shadows? the only logic I can see relevant to this would be that they are mapped pixels and that the interpolation isn't done on the final rendering of the image ie the surrounding pixels are dark, the mapped ones interpolated black then subsequently there is light delivered to the area but the mapped areas stay black.....probably overthinking this!

jmunkki
6th November 2013, 01:06 PM
It's likely that the camera is taking multiple shorter exposures and then either applying noise reduction on the sum of those exposures or on individual exposures that are accumulated into a single longer one. Or maybe they aren't applying noise reduction when live bulb mode is used?

I don't see this as a sensor issue that Olympus could fix in production (unless there's a huge sensor tech breakthrough). Could they do better with the noise reduction? Possibly/probably. I wouldn't be surprised if they could improve image quality in live bulb mode with a firmware update to make the noise reduction more accurate.

If you want to understand the behavior, I would try to test and see if the black pixels you see correspond to hot or cold pixels on the sensor. If they are hot pixels, then the noise reduction is overcompensating in live bulb mode. If they are cold pixels, then the noise reduction isn't compensating enough.

http://www.cyanogen.com/help/maximdl/Dark_Frame_Calibration.htm

andym
6th November 2013, 01:15 PM
As it's crap weather and I'm bored and still have a cold, thought I'd have a play.Not used live time before so I might also learn something.

This is f8,9.8second exposure using Livetime, ISO1600 at 60mm on the 12-60.

Just resized
http://www.e-group.uk.net/gallery/data/500/PB060836resize.jpg (http://www.e-group.uk.net/gallery/showphoto.php?photo=67374)

Centre crop
http://www.e-group.uk.net/gallery/data/500/PB060836crop1.jpg (http://www.e-group.uk.net/gallery/showphoto.php?photo=67372)

Top LHS crop
http://www.e-group.uk.net/gallery/data/500/PB060836crop2.jpg (http://www.e-group.uk.net/gallery/showphoto.php?photo=67373)

No other PP. Sorry but I can't see anything amiss. I still think you may have a sensor issue.

Ps exif intact also

PPS noise reduction turned off so no dark frame subtraction.

jmunkki
6th November 2013, 01:39 PM
I think birdboy only had a problem where near-dark pixels (black sky) went completely black. Andym's test photos don't have much black in them - they look great though.

birdboy
6th November 2013, 01:43 PM
I'll see what I can do - but from my understanding of your previous post what we are really looking at is "what's the camera doing differently when shooting LT and bulb?"

then whatever that difference is must, in some way, be delivering those black dots.

are we on the same wavelength?

btw why should they be specific to the shadows? the only logic I can see relevant to this would be that they are mapped pixels and that the interpolation isn't done on the final rendering of the image ie the surrounding pixels are dark, the mapped ones interpolated black then subsequently there is light delivered to the area but the mapped areas stay black.....probably overthinking this!

Thanks I haven't tried bulb but dialled in shutter speed and I can see a difference.

I think the dots in the shadows must relate to the intensity of the light in the photocell and the threshold that gives it an output that the camera can work on for its picture processing if that makes any sense. I get the black dots with raw files so I would not expect in camera processing to any difference.

In order to see this effect it is really important that your picture contains shadowed areas of lower light intensity, hence it really shows up in night exposures and fireworks.

birdboy
6th November 2013, 01:47 PM
It's likely that the camera is taking multiple shorter exposures and then either applying noise reduction on the sum of those exposures or on individual exposures that are accumulated into a single longer one. Or maybe they aren't applying noise reduction when live bulb mode is used?

I don't see this as a sensor issue that Olympus could fix in production (unless there's a huge sensor tech breakthrough). Could they do better with the noise reduction? Possibly/probably. I wouldn't be surprised if they could improve image quality in live bulb mode with a firmware update to make the noise reduction more accurate.

If you want to understand the behavior, I would try to test and see if the black pixels you see correspond to hot or cold pixels on the sensor. If they are hot pixels, then the noise reduction is overcompensating in live bulb mode. If they are cold pixels, then the noise reduction isn't compensating enough.

http://www.cyanogen.com/help/maximdl/Dark_Frame_Calibration.htm

As I understand Olympus's own information these black dots are not hot pixels that is another issue see the following
http://www.olympusamerica.com/crm/on..._oly_03_09.asp

andym
6th November 2013, 01:49 PM
It did occur to me that this might occur more in shadow areas so I tried this.

http://www.e-group.uk.net/gallery/data/500//PB060837resize.jpg (http://www.e-group.uk.net/gallery/showphoto.php?photo=67376)

http://www.e-group.uk.net/gallery/data/500/PB060837crop2.jpg (http://www.e-group.uk.net/gallery/showphoto.php?photo=67377)

http://www.e-group.uk.net/gallery/data/500/PB060837crop3.jpg (http://www.e-group.uk.net/gallery/showphoto.php?photo=67378)

birdboy
6th November 2013, 01:51 PM
As it's crap weather and I'm bored and still have a cold, thought I'd have a play.Not used live time before so I might also learn something.

This is f8,9.8second exposure using Livetime, ISO1600 at 60mm on the 12-60.

Just resized
http://www.e-group.uk.net/gallery/data/500/PB060836resize.jpg (http://www.e-group.uk.net/gallery/showphoto.php?photo=67374)

Centre crop
http://www.e-group.uk.net/gallery/data/500/PB060836crop1.jpg (http://www.e-group.uk.net/gallery/showphoto.php?photo=67372)

Top LHS crop
http://www.e-group.uk.net/gallery/data/500/PB060836crop2.jpg (http://www.e-group.uk.net/gallery/showphoto.php?photo=67373)

No other PP. Sorry but I can't see anything amiss. I still think you may have a sensor issue.

Ps exif intact also

PPS noise reduction turned off so no dark frame subtraction.

As has already bee pointed out I would not expect to see the black dot effect in these pictures because the expose is evenly lit.

I was really hoping that folk may have so examples of trying to use live time for night shots or light trails (but not stair trails)

But thanks for your interest and testing Andy

birdboy
6th November 2013, 02:01 PM
It did occur to me that this might occur more in shadow areas so I tried this.

[/img][/url]


It is very specific areas of the picture where this manifests itself. Such as this

http://e-group.uk.net/gallery/data/500/Black_dots_Livetime.jpg (http://e-group.uk.net/gallery/showphoto.php/photo/67352)

Andy any chance of pm that picture for me to have a look. Thanks John

andym
6th November 2013, 02:08 PM
Hmmm, I've had a play but not sure how to connect a full size image to a PM if that's possible.:o:o:o

If not PM me an email address and I'll send you a full size copy.

photo_owl
6th November 2013, 02:32 PM
As I understand Olympus's own information these black dots are not hot pixels that is another issue see the following
http://www.olympusamerica.com/crm/on..._oly_03_09.asp

I think you are missing what jmunkki is saying

what you are experiencing probably has it's root in hot or cold pixels, but it's not simply their impact in normal shooting - effectively as Andy's illustrated here and you have seen for yourself with the timer shot.

Livetime is creating it's image a whole different way (the display frequency is shown in the menu and differs for different ISO settings - which also seems to tie in with your stated beliefs re sensitivity and dark areas) - but what other processes are affected in the final image presentation using Livetime are unknown (as per my original post). It's in there you will find your answers - but even Oly will probably have to dig!

The more I play with it the more I don't think it's really creating the image differently, only the display element whilst shooting. There may be an impact on the system as a whole to both record and display the image at the same time ie working harder, but that's about all.

birdboy
6th November 2013, 02:58 PM
Hmmm, I've had a play but not sure how to connect a full size image to a PM if that's possible.:o:o:o

If not PM me an email address and I'll send you a full size copy.

Thanks Andy got your attachments the raw file clearly shows the black dots.

http://e-group.uk.net/gallery/data/500/AMBD1.jpg (http://e-group.uk.net/gallery/showphoto.php/photo/67379)

Not sure what jpg settings you were using but could not see them in your jpg:confused:

I use LSF for jpgs.

andym
6th November 2013, 02:59 PM
I'm a little baffled with the image you sent me. The black dots seem to get worse the further to the left you go.
Along the RHS apart from the very bottom I can not see them at all, for example.

Crop above the tree line in the centre
http://www.e-group.uk.net/gallery/data/500/PB020053crop1.jpg (http://www.e-group.uk.net/gallery/showphoto.php?photo=67380)

Crop above the trees above the tree line
http://www.e-group.uk.net/gallery/data/500/PB020053crop2.jpg (http://www.e-group.uk.net/gallery/showphoto.php?photo=67381)

To my old eyes they fairly close in brightness.

Draw you own conclusions but I would expect them both to have the dots if its the processing Livetime does.

andym
6th November 2013, 03:13 PM
Right here you go.

Same crop as your more or less processed my normal way ie Converted to a full size jpeg in Silkypix and then resized in Elements 10.
http://www.e-group.uk.net/gallery/data/500/PB060837crop5.jpg (http://www.e-group.uk.net/gallery/showphoto.php?photo=67382)

Don't use Viewer but this is processed through it.

http://www.e-group.uk.net/gallery/data/500/PB060837viewer.jpg (http://www.e-group.uk.net/gallery/showphoto.php?photo=67383)

Look at those dots.

Curiouser and curiouser

andym
6th November 2013, 03:21 PM
It's visible in my test if processed through Viewer.

http://www.e-group.uk.net/gallery/data/500/PB060836viewer.jpg (http://www.e-group.uk.net/gallery/showphoto.php?photo=67384)

So is it a processing issue?

andym
6th November 2013, 03:23 PM
John

Have you got an ORF of the lake shot,or one of the others that show the problem that you can mail to me?So I can have a play.

Ian
6th November 2013, 03:40 PM
As I understand Olympus's own information these black dots are not hot pixels that is another issue see the following
http://www.olympusamerica.com/crm/on..._oly_03_09.asp

Link is u/s.

Ian

Ian
6th November 2013, 03:42 PM
Two things come to mind - have you switched off Noise Reduction (not to be confused with Noise Filter, which is unrelated)?

Secondly, have you tried running the Pixel Mapping routine?

Ian

photo_owl
6th November 2013, 04:54 PM
I've just run a number of shot using Livetime and normal M shutter with exposures between 30" and a minute.

Delay was in getting Oly Viewer to import (this machine is way under spec) but l cannot discern any difference in what the camera JPEG (Ln) and the viewer rendered RAW file at the maximum magnification available.

Which makes me feel 'left out' again.

Will run more over the next few hours with clearer backgrounds (now that I see Andy's file delivered in the light areas too.....)

Right, run more, and now have some black dots (with a white halo)....in both renderd Orf and the jpeg.


http://farm3.staticflickr.com/2871/10711129006_1ae932a371_o.jpg
50x38 crop @ 1:1

Ran pixel mapping

Re-shot exactly the same shot (never moved camera), and I still have black dots.

Now trying to establish if they are the 'same dots'....

birdboy
6th November 2013, 05:44 PM
I'm a little baffled with the image you sent me. The black dots seem to get worse the further to the left you go.
Along the RHS apart from the very bottom I can not see them at all, for example.


To my old eyes they fairly close in brightness.

Draw you own conclusions but I would expect them both to have the dots if its the processing Livetime does.

Thanks for help on this Andy I was beginning to think that there was something wrong with just my camera.

I am more than a little baffled:eek:. My only explanation for the increase in black dot density is that the light value is lower albeit only slightly. I have interpreted this as lazy pixels because they are not dead when there is more light shinning on them.

There must be something to do with the in camera processing of raw data. During live time the camera is also developing the image for display on the screen and I just wonder if some pixels / photosites miss getting processed (analogue to digital conversion) because the camera processor is doing the display bit. Just my crazy thinking.

birdboy
6th November 2013, 05:46 PM
John

Have you got an ORF of the lake shot,or one of the others that show the problem that you can mail to me?So I can have a play.

Sorry Andy only took these LSF as I use LR4.4 for post processing.

birdboy
6th November 2013, 05:48 PM
Link is u/s.

Ian

Whoops something wrong with my copy/paste skill.:o Thanks for letting me Ian.

Try this:

http://www.olympusamerica.com/crm/oneoffpages/ask_oly/crm_e_ask_oly_03_09.asp

Ian
6th November 2013, 05:50 PM
Does Noise Reduction make any difference (dark frame subtraction).

Ian

I've just run a number of shot using Livetime and normal M shutter with exposures between 30" and a minute.

Delay was in getting Oly Viewer to import (this machine is way under spec) but l cannot discern any difference in what the camera JPEG (Ln) and the viewer rendered RAW file at the maximum magnification available.

Which makes me feel 'left out' again.

Will run more over the next few hours with clearer backgrounds (now that I see Andy's file delivered in the light areas too.....)

Right, run more, and now have some black dots (with a white halo)....in both renderd Orf and the jpeg.


http://farm3.staticflickr.com/2871/10711129006_1ae932a371_o.jpg
50x38 crop @ 1:1

Ran pixel mapping

Re-shot exactly the same shot (never moved camera), and I still have black dots.

Now trying to establish if they are the 'same dots'....

birdboy
6th November 2013, 05:54 PM
Two things come to mind - have you switched off Noise Reduction (not to be confused with Noise Filter, which is unrelated)?

Secondly, have you tried running the Pixel Mapping routine?

Ian

I have gone through a number of options regarding the noise reduction and noise filtering. I have tried
NR NF
off off
on off
on low
on standard
on high
Auto off
Auto low
Auto standard

I do not think is a noise issue because noise is random. These black dots are consistent in where they appear when they appear.

Yes I have run pixel mapping 3 times in total since I have had the camera.

birdboy
6th November 2013, 06:00 PM
It's visible in my test if processed through Viewer.

So is it a processing issue?

Not sure it is because if you take the same picture but use a fixed shutter stop there are no black dots. If its processing then its in camera.

I am also wondering that it maybe that some software takes out these black dots as part of their cleaver processing. Clearly LR 4.4 or Viewer 3 does not.

Ian
6th November 2013, 06:05 PM
I have gone through a number of options regarding the noise reduction and noise filtering. I have tried
NR NF
off off
on off
on low
on standard
on high
Auto off
Auto low
Auto standard

I do not think is a noise issue because noise is random. These black dots are consistent in where they appear when they appear.

Yes I have run pixel mapping 3 times in total since I have had the camera.

Noise Reduction deals with non-random noise; i.e. stable inconsistencies due to stuck or hot pixels; so after your photo is taken the sensor is energised for the same time as the previous exposure and then any hot pixels are mapped out using data from surrounding pixels. The repetition of the exposure time will ensure any pixels affected by the time the sensor is sensitised will be detected.

Pixel mapping will detect pixels that are stuck at 100% (white) or (black) 0% and use neighbouring data to hide them. This might help with your problem. Sensors tend to get more faulty pixels with age so it's a good idea to run the pixel mapping routine periodically.

Ian

birdboy
6th November 2013, 06:19 PM
Noise Reduction deals with non-random noise; i.e. stable inconsistencies due to stuck or hot pixels; so after your photo is taken the sensor is energised for the same time as the previous exposure and then any hot pixels are mapped out using data from surrounding pixels. The repetition of the exposure time will ensure any pixels affected by the time the sensor is sensitised will be detected.

Pixel mapping will detect pixels that are stuck at 100% (white) or (black) 0% and use neighbouring data to hide them. This might help with your problem. Sensors tend to get more faulty pixels with age so it's a good idea to run the pixel mapping routine periodically.

Ian

Thanks Ian for the explanation regarding non-random /hot dead pixels I had not considered this aspect because I had assumed that the pixel mapping would take them.

The Oly advice seems to indicate doing pixel mapping once a year and I would hope that we are not talking about many pixels. I have very crudely estimated that the black dots I am seeing goes into the thousands at that rate it will not be long before my 16Mpixel camera become a 2 Mpixel camera:eek:

It seems to me that there is something in the camera processing that is not processing some pixels hence they get the value of 0. The fact that others have confirmed pixel mapping makes no difference suggests to me that the effect is not hot or dead pixels because when pixel mapping is carried out they behave as they should.

Ian
6th November 2013, 07:14 PM
I think you said that pixel mapping didn't make a difference so it's probably an image processing glitch rather than stuck pixels.

Ian

andym
6th November 2013, 07:20 PM
I think you said that pixel mapping didn't make a difference so it's probably an image processing glitch rather than stuck pixels.

Ian


Yes it's got to be, my examples are raw out of the camera not converted to jpeg in camera. If I run them through my normal workflow there's no dots, if I convert them to jpegs in Viewer I get the dots as shown earlier.
I haven't tried converting a raw to jpeg in camera to see what happens. I might try later.

Best thing is it appears not to be your camera John.

photo_owl
6th November 2013, 07:24 PM
Does Noise Reduction make any difference (dark frame subtraction).

Ian

NR is on (by default)

I would expect to see a few 'bright spots' without it, but will check

birdboy
6th November 2013, 07:28 PM
I think you said that pixel mapping didn't make a difference so it's probably an image processing glitch rather than stuck pixels.

Ian

Yes that's right there was no change. If it is an image processing glitch then it is having very many of them during one frame.:eek: I am hoping that this turns out to be a coding bug fixed by a new release of firmware. But now that we have seen that it is not just specific to my camera lets hope Olympus take this issue and investigate it.

Am I being too fussy when pixel peeping. I had some friends come round this afternoon and I showed them the lake picture on my TV. They did not see the black dots until I pointed them out then they could see them. At x 2 they were clearly visible.

photo_owl
6th November 2013, 07:35 PM
Yes it's got to be, my examples are raw out of the camera not converted to jpeg in camera. If I run them through my normal workflow there's no dots, if I convert them to jpegs in Viewer I get the dots as shown earlier.
I haven't tried converting a raw to jpeg in camera to see what happens. I might try later.



I shot RAW + jpeg and the jpeg is identical to the Viewer rendered orf in this situation ie same dot's magnitude and position.

My shot was a slab of marble, and there were no large darker areas ie it was pretty flat with small local changes. Probably about 10-15 black dots on the frame reasonably evely, but randomly, distributed.

Same camera settings, same position and exposure but with a cyclemen in front got me 2 dots total (that I could find when I went back and looked again;harder....)

not a cause of worry for me, but interesting none the less

andym
6th November 2013, 08:56 PM
My last test, if you convert an ORF shot using livetime to a digital negative with DNG converter and then open it in photoshop(I assume LR would be the same)there are no dots.Just like my Silkypix conversion.
To me it points at something Viewer is doing.

birdboy
6th November 2013, 09:18 PM
My last test, if you convert an ORF shot using livetime to a digital negative with DNG converter and then open it in photoshop(I assume LR would be the same)there are no dots.Just like my Silkypix conversion.
To me it points at something Viewer is doing.

I do not have DNG converter or photoshop. I would guess that photoshop is doing some hidden processing (removing black dots) and that's why you do not see them. The software that is supplied with the camera is Viewer 3.

Anyway I see the black dots on the OM1 screen when viewing so its not related to post camera software processing.

Thanks for your work on this Andy but I think we should wait and see what Olympus has to say now.

andym
6th November 2013, 09:31 PM
Ahh, but what you see on the screen is a jpeg after decoding in the camera.

By the way DNG converter is free if you want to try it. I think both LR and Photoshop use ACR so there should be no difference.I can see the blobs with the conversion from Viewer but not with my other methods and am only using Photoshop to resize it for web.

I'll be interested if you get any meaningful response for Oly support.

PeterBirder
6th November 2013, 10:13 PM
FWIW I have for some time been of the opinion that the Viewer software is essentially the same as the in-camera raw conversion firmware. The results of Andy's tests to me confirm this view.

From this I would conclude that your "black dot" problem is due to the camera firmware/Viewer software creating them rather than other raw conversion programmes removing them.

Regards.*chr

birdboy
6th November 2013, 10:28 PM
FWIW I have for some time been of the opinion that the Viewer software is essentially the same as the in-camera raw conversion firmware. The results of Andy's tests to me confirm this view.

From this I would conclude that your "black dot" problem is due to the camera firmware/Viewer software creating them rather than other raw conversion programmes removing them.

Regards.*chr

Well I do hope you are right. As it appears that you get these with the EM5 that does seem to add to your thinking.

Thanks to everyone for contributing on this I will keep you informed of any progress I get from Olympus tech support.

John
Ps I still think this is great little camera with great potential for some very creative photography.

birdboy
6th November 2013, 11:57 PM
Ahh, but what you see on the screen is a jpeg after decoding in the camera.

By the way DNG converter is free if you want to try it. I think both LR and Photoshop use ACR so there should be no difference.I can see the blobs with the conversion from Viewer but not with my other methods and am only using Photoshop to resize it for web.

I'll be interested if you get any meaningful response for Oly support.

Did not understand this the first time round but have downloaded the DNG converter taken some more raw pics converted them into DNG and then imported them into LR4.4 and guess what no black dots. Viewed the raw file in Viewer guess what black dots.

So Peter it looks like you are right the in camera jpg conversion looks to be the same as Viewer 3.

My advice for now then for those wanting to use livetime is to use raw files and do not use Viewer 3 to process them until Olympus fix this.

birdboy
9th December 2013, 05:33 PM
An Update which may be of interest to others taken longish shutter times.

As a recap I started this thread because I took firework display shots with NR ON circa 6 secs in LSF only (no raw) and on some of my pictures I got many black dots. I was able to reproduce this effect in my study.

I have been in contact with Olympus helpline with emailing them my files.

What I have now been finding is that with the new firmware 1.1 when I took raw + LSF I could not repeat the black dots in either file. But if I turned NR OFF I got black dots in the LSF jpeg but not the raw file!:confused: More testing and I got black dots in raw + LF but not with raw + LN! I asked Olympus about this;

"This is the other file taken at 13 secs (PB281456.jpg) (attached) which was taken RAW + LSF with NR off. The raw file shows no black dots but this LSF shows black dots. This was taken to show that taken with the same shutter speed and at RAW +LSF you get black dots in the jpeg but not the raw. Showing to me that there is a problem with the camera jpeg process adding black dots.

Does this make sense or can Olympus explain this?"


Their reply is clear enough but does not make sense to me;

"Yes this is normal. In order to eliminate black dots you need to switch the NR on. With NR off the LSF will show black dots."


So it seems that it is normal to get black dots with the LSF file with NR OFF. I have to then question what's the point of shooting raw + LSF or even using just LSF for long exposures. I have now changed my setting when needed to shoot raw + LN (although I have gone back to shooting in raw only mostly) as this seems to get round the problem.

Its not the answer I wanted and I doubt I can trust shooting in LSF or LF alone.

I also have left the NR in Auto but Olympus have not told me (despite being asked) at what shutter speed the black frame subtraction kicks in.

Not sure if this will help anyone but I thought I would share the answer.

Zuiko
9th December 2013, 06:09 PM
Thanks for sharing this, John but it does sound rather random and lacking a definitive explanation. Sorry that it's still not resolved but it does seem to underline the complexity of camea software, no wonder I'm often baffled. At least you now seem to have a work-around.