The first question: SmartClip - how to handle clipped highlights from raw

[AlexisMagni]

After Roberto excellent words on how powerful ColorPerfect is with Digital Images as well as Film Scans, I decided to throw at it some diversity of photos in different lighting I have on the HD.

I am impressed with the natural rendition of colors, it pleases me more than any of the Embedded Profile from Nikon. But I am facing an issue that I might need some help on.

The issue begin to happen when I have Blown Out Highlight on the Photo. Usually, in daytime, it will be the sun, that will always be clipped no matter what. But in this night scene, the clipped area comes from a much larger lighting setup from this showcase.

This is the original image as interpreted originally by Nikon Flat Profile:


Now, after generating a Linear .Tiff using MakeTiff, with the embedded AdobeRGB Profile, and conversion, I face the issue specially on the blown out "angel wings". If I dont have Smart Clip activated, I have Magenta Tint on the Highlight. In order to avoid the magenta tint, I have to increase the "Black" until it is pure white, this however leads to a loss of detail in a lot of the highlights. If I decrease the Highlight to recover the detail, I get more and more magenta.


If I enable Small Clip however, which should fix the Magenta Cast, then I have artifacts around the Angel Wing, as shown below:


Anything else I can try? I can host the Raw if needed

Thanks!

[C.Oldendorf]

Alexis, hi, great question. That takes us right to the more complex stuff

The general write-up on SmartClip can be found here:
https://www.colorperfect.com/smart_clip.html?lang=en

In short:
Overexposure is never perfect. Different parts of an image can fail differently because of off-balance capture that is intrinsic to digital cameras. It’s essentially “garbage in, garbage out” — but SmartClip gives us a way to fix it. Many imaging solutions make these areas devoid of color and use gradients to include adjacent brightnesses gradually. In the capture, it is a hard failure in data continuity.

How to use SmartClip in practice

• SmartClip lets you tell the system what color an object should have.
• If you need to correct multiple objects, you can do this by blending edits. With two, for example, set up a selection that divides the image in half.
• In ColorPerfect, after all general adjustments, tick Use Selection and work on In and Out separately.
• For each area:
  - Next to SmartClip, press SC Freeze.
  - Click in the vicinity of the defect as many times as needed until the color looks decent.
  - Then press SC Lock to lock it in and continue editing.
• Switch to Out (the other half of your selection) and repeat for the second overexposed area.

This way, SmartClip takes the recovered brightness of what was previously an “unknown” color and assigns it the correct color you want.

[Full image link - opens in new tab]

[Full image link - opens in new tab]

If we have more than two areas to differentiate, just make a few copies of the RAW as layers. After editing the first, use "Exact" value carry-over on the next one as detailed here: https://www.colorperfect.com/restoring/settings/from/previous/use/of/ColorPerfect/ That is in fact one of the few sensible uses of "Exact". After that, quickly set your SmartClip color, OK/Out, and so forth. Starting on the top image, use the eraser tool in PS to punch through to the underlying image where that was corrected for highlights. Ctrl+E (Win) / Cmd+E (macOS) to merge the two layers and continue with the one underneath. Sounds way more complicated than it is in practice. Generally, I often shoot at –0.3 or even –0.7 EV and save myself the trouble, but it's not always possible to avoid overexposure. The elephants were simply exposed wrong, but the tent is far out... Specular highlights are often just the color of the light source (i.e. neutral) and usually need no special treatment, apart from SmartClip being active — which it is by default.

I should probably mention that the tent was reined in, brightness-wise using ColorPerfect's highlight compression. Of course we have way less latitude in digital captures than we do in color negatives but this is the general idea of that: https://www.colorperfect.com/highlights.html?lang=en With SmartClip used on color that color will be used in recovered areas as well.

[robyferrero]

Dear Alexis,
I was waiting for you
I knew this would be one of the first problems you'd encounter.

It's easily solved by learning the adjustment method. Although I must say that in some cases I had some difficulties.
I don't know why—Christoph can explain—but in some cases, using SC Freeze + clicking on adjacent areas, I wasn't able to achieve any changes, no color applied.

I'll take this opportunity to reread all the relevant information.

[C.Oldendorf]

in some cases, using SC Freeze + clicking on adjacent areas, I wasn't able to achieve any changes, no color applied.

SmartClip does reject pixels that are too close to clipping — once any channel is maxed out, the ratios aren’t reliable anymore. That’s why clicks near blown areas may get ignored. If you can find the same color a bit darker, it works just as well since SmartClip uses the hue, not the brightness, of the sample.

[AlexisMagni]

It seems everyone was waiting for me to show up with these questions!

Thank you for the explanations! A few things I want to clarify more as I pick more images to try and work on the fixes and adjustments.

- When you mention the SC Freeze, this is with SmartClip enabled? It doesnt have any functionality while SmartClip is disabled right? Because for example, I only have the Magenta Hue when SmartClip is deactivated, as soon as I click it on, the Magenta turn back into white and I no longer need to correct color. What I do have are the artifacts around the blown out highlights, what would work best for this picture would be a mix of SmartClip Magenta Fix, with the tones and smooth gradation of SmartClip deactivated, which shows no artifacts, just the wrong Magenta Cast. It is an extremely overexposed highlight I know This picture was a long exposure and the dynamic range between dark and bright areas are beyond what any sensor would be able to work with.

- I couldnt find the In and Out options you talk about? Where do I find those?

Regards!

[robyferrero]

SmartClip does reject pixels that are too close to clipping — once any channel is maxed out, the ratios aren’t reliable anymore. That’s why clicks near blown areas may get ignored. If you can find the same color a bit darker, it works just as well since SmartClip uses the hue, not the brightness, of the sample.

Christoph,

I tried more distant areas, but probably not darker. Good.

It seems everyone was waiting for me to show up with these questions!

Alexis,

funny and nice


Christoph will answer you correctly.

In the meantime:

SmartClip + SC Freeze to fill the color.
You can click the adjacent area several times to choose the best shade.
You can also press the SC Freeze button again to start over from scratch in the color search.
Now, in your specific case of the wings, they still look white to me; I imagine they're illuminated by lamps.
So they're wing-shaped lights, no detail, no color; in this case, SC Freeze + click the adjacent color isn't necessary. You just need a SmartClip in that specific area.

About the artifacts, well, I don't know what to say.
Let's wait for Christoph.
If I had to do it my own way, empirically and laboriously, I'd use a first image with SmartClip disabled. Then a second image with SmartClip enabled, and by selecting just the wings, I'd merge the second image with the first.
This way you exclude the artifacts on the wings, and you exclude the artifacts in the other areas.
Or, perhaps, the inside-out selection.

When you use SmartClipp + SC Freeze + click on an adjacent color, be careful because that color is transferred to all the magenta clips. So if you sample yellow, but you have another magenta clip on the blue, that blue becomes yellow.

Also, empirically and laboriously, you can create two different files, one with the yellow correction, and the other with the blue correction, and then merge them into a single file.

Or, I think, the inside-out selection.

As for the selection, I don't remember how to do it because I never use it.

It seems to me that you need the alpha channel, but I don't want to be wrong.

Let's wait for Christoph.

[C.Oldendorf]

The matter of overexposure on a sensor is a tricky one. To make it easier to picture, I prepared a graphic that shows what happens inside the silicon itself. Of course, real sensors are more complicated — they have microlenses, color filter stacks, and much more. But if we simplify, each little square under the mosaic is a sensor well. Each well is a collector of photons. A useful way to imagine it is like a vessel that fills with liquid: photons pour in like drops of water. Each vessel has a capacity. Once it is full, it cannot tell us how much more light was really there.

[Full image link - opens in new tab]

Now, in practice, cameras avoid giving us the “messy part” of that story. The analog-to-digital converters only digitize a range where the signal is reliably linear. So in our data, there is no slow bending away from linearity — just a hard stop. What we receive as raw values are like the liquid gauges shown in the graphic.

ColorPerfect works in a very pure way: each pixel is treated on its own, one by one, without reference to its neighbors or the geometry of the scene. So if we look at one example pixel, we may find that red and blue are measured correctly, but green has spilled over. The well is full. In histogram terms, those pixels are the spike at the right edge.

Here comes the problem: when green is maxed out, we no longer know how much more should have been there. Was it just one extra drop? Was it double or triple? We cannot know. What we do know is that something is missing. And the absence of green eventually appears as its complementary — magenta.

This does not show itself all at once. With SmartClip off, the defect grows gradually: a little missing green may only dull the color, while more missing green drifts into magenta. The information is already unreliable, but the eye only notices once the error becomes large.

When Dave and I first designed PerfectRAW, this problem demanded an answer. That is when SmartClip was born. It is not a magic cure, but a principled choice: do not invent color where the signal is broken. Instead, take brightness from the data, and allow the user to guide the missing color. That is what SmartClip lets you do. With a click, you tell ColorPerfect: “This should have been blue,” or “This should have been yellow.” The program then uses that direction to restore harmony.

It is possible that one day we may find new solutions here. But for now, SmartClip remains as it is, and it has proven reliable.

Returning to Alexis’s image: why do we see artifacts on the blue wall, or along the edges of the wing-shaped lamps? It is exactly this: one channel has clipped and obviously barely so - hence the "artifacts", so the program refuses to trust the color. With SmartClip active, a proper click on the blue wall and another on the yellow screen will set things right. But for the lamps themselves, which are meant to be pure white light, the correct treatment may require some touch-up in Photoshop, at least where they overlap the yellow screen.

The easy answer is don't overexpose, only that that can be hard

[C.Oldendorf]

- I couldnt find the In and Out options you talk about? Where do I find those?

First we set up a selection in PS...

Selection_in_PS.jpg (95.87 KiB) Viewed 964 times

[Full image link - opens in new tab]

...then in ColorPerfect we get Use Selection...

[Full image link - opens in new tab]

... and we can do just that, use it, to adjust two halves (even with gradients loaded as selections).

[Full image link - opens in new tab]

To illustrate I just added 4EV of Black to the In part of the selection. Of course we talked about adjusting SmartClip differently for In and Out but that would not stand out so boldly.

[C.Oldendorf]

When you mention the SC Freeze, this is with SmartClip enabled? It doesnt have any functionality while SmartClip is disabled right? Because for example, I only have the Magenta Hue when SmartClip is deactivated, as soon as I click it on, the Magenta turn back into white and I no longer need to correct color.

Your image has three areas of color failure due to overexposure. The wings look fine at SmartClip’s default of making them neutral under the current CC filter pack. That is what is “smart” about it — if you change color balance, the clipping points will follow. But the blue wall looks like it needs color reassigned in its overexposed region, as does the yellow screen. The latter may be the most elaborate edit to blend with the version where the wings are neutral, as you will find.

It seems everyone was waiting for me to show up with these questions!

I didn’t know before the fact, but I am glad you did, because it is a situation deeply rooted in digital imaging and generally hidden from the user. So this whole business of colors being rejected for rendering versus gradients toward magenta color casts can be disconcerting for anyone who comes across it the first time. That even goes so far that people tell me MakeTiff does not extract an unadulterated equivalent of the pixel data in the raw image — that something must be wrong or missing, because some other software conjures up this and that.

Ah, “conjures” — a beautiful place to quote Dave from an old email I have before my mind’s eye, if I can find it… Yes, adjacent subject, but epic enough a quote that I knew what phrase to search for 14 years later.

That is incredible. It appears that it was thoroughly screwed up from day one and Adobe just added to it later. It already was pure sorcery. If it isn't doing what we want yet, add the tried and true wing of bat and eye of newt.

[robyferrero]

This is one of maybe two photos I processed with Color Perfect where Smart Clip didn't work.
Could it be that in this case, assuming the exposure is really too high, I need to change the default Smart Clip value in the options?

The two examples show the before and after of highlight recovery.

[C.Oldendorf]

The two examples show the before and after of highlight recovery.

Could you make a downsized version of the linear TIFF that goes into this — say around 1200 px wide? Then I’ll explain what’s going on with it.

[robyferrero]

Here is the reduced version of the linear Tiff.

[C.Oldendorf]

Yes, that we can look at. For whatever reason, the cutoff point falls within the data. SmartClip, in its current implementation, does not account for that.

cut_off_moved.jpg (165.04 KiB) Viewed 926 times

[Full image link - opens in new tab]

It’s fine to use Levels on RGB (but only the Highlight slider!) on the linear data before ColorPerfect and PerfectRAW when you see something like this — the problem will disappear.

After that, SmartClip in conjunction with Highlight Compression and color definition can do its work.

[Full image link - opens in new tab]

We do have two distinct colors here, so selections as we discussed are possible, and there is also a luminance threshold at the cutoff that needs retouching. This can happen because the brightness estimation is, after all, only an estimation — and in this case reality must be somewhat darker.

If we want to look at why that cutoff point is where it is, we'd need to go on with the raw file. We can.

[robyferrero]

Can we use PS levels on RGB highlights, or just on green and blue?

Here's the RAW file.

[C.Oldendorf]

Can we use PS levels on RGB highlights, or just on green and blue?

Glad you asked — the answer is that we can NEVER do that. The reason is simple: brightness on linear RGB is equivalent to exposure. Adjusting any individual channel would effectively alter the sensor’s response characteristics, and afterwards PerfectRAW would no longer have a valid characterization to work with. Pre-processing must therefore be handled with the utmost care.

Edit: I looked at the raw image and there is no trivial reason in the metadata that should have been heeded. Cute quirk.

[robyferrero]

The mysteries of digital

I don't even know what happened in that shot.
Blur aside, the rest should be as I usually do.
It looks like a shot that slipped out of the camera.

[robyferrero]

Here I have another problem.

In this photo, SmartClip isn't working, and I don't know how to pre-correct in PS.

2014-100-2624-SmartClip_magenta-roby_ferrero.jpg (361.93 KiB) Viewed 315 times

[Full image link - opens in new tab]

As per the attached example, I tried using the tonal values ​​from PS directly on the Linear TIFF from MakeTiff.

2014-100-2624-Linear_Tiff-SmartClip_magenta-crop_3-roby_ferrero.jpg (382.68 KiB) Viewed 315 times

[Full image link - opens in new tab]

A little more, a little less, but it didn't work. I don't know what to do or which tool to use in PS for this problem.

2014-100-2624-SmartClip_magenta-crop_2-roby_ferrero.jpg (497.15 KiB) Viewed 315 times

[Full image link - opens in new tab]

2014-100-2624-SmartClip_magenta-crop-roby_ferrero.jpg (452.25 KiB) Viewed 315 times

[Full image link - opens in new tab]

[C.Oldendorf]

For Smart Clip it is required that the overexposed image areas actually touch the right end of the histogram. There have been situations where, after interpolation, that was not the case in images I’ve seen. Most often, this was related to dcraw doing the interpolation. I don't know about libraw, but I suppose it might behave the same. Generally, I have not seen this when DNG Converter is used for interpolation — or at least I don't recall encountering it. So my first question would be: what was this image interpolated with? Can we determine what happens at that stage? Only thereafter would I look at fixing the problem in Photoshop, which is basically about making the overexposed areas touch the right edge of the histogram using the highlight slider in Levels — not the input slider below, but the actual highlight slider above of where you drew your red arrow.

Looking at the actual image would be helpful to puzzle this out.

[robyferrero]

I had a good laugh when I discovered that the levels slider was supposed to be used the other way around.

The RAW file was passed into DNG Converter using MakeTiff in AdobeRGB1998.

Now I'll try setting the histogram to 255 and see what happens.

I'm attaching the RAW file.

[robyferrero]

I can already confirm that by bringing the PS histogram to precisely 255, not even one more click, SmartClip works, just like SC Freeze does.

[C.Oldendorf]

Great, I will look at your image later and see whether I can find out anything about why this might happen — by examining its metadata and trying a few parametric interpolation alternatives.

In the meantime, I can explain why SmartClip does not engage if the highlights do not touch the right edge of the histogram. It’s actually very straightforward. Suppose you have a wall the color of vanilla pudding, a creamy yellow shade — and this wall has been exposed perfectly. It just falls short of saturation in the green channel of your raw capture by, say, 5% before the right edge.

If SmartClip were to simply search for the brightest portion of the histogram and assume that region to be overexposed, your cream-colored wall would initially lose its color. Yet the color information is fully intact in the capture. So the only reliable symptom of clipping in the raw data that we can use is the very last histogram bin — or, since there are effects that can shift this slightly, a very narrow group of bins at the far right edge. That is the current implementation.

This is also why everything behaves beautifully after you use the level slider, as we discussed: hold the Alt key (Option on Mac) while moving the slider, and Photoshop will show overexposed pixels emerging from a black preview. Stop when you first see pixels, then add a numerical +1 again so that none show — and that should be sufficient.

Of course, I would still prefer to reason this out using metadata and similar information, because this behavior is clearly not intended. Speaking of this, I remembered a raw file a friend sent me recently, shot on an Olympus E-P1 and underexposed by at least three stops — yet the highlights still turned magenta. That suggests something happening at the sensor level or in the analog-to-digital conversion chain, which is unusual. I may ask him to share the file here if he is willing.

[robyferrero]

Well, I double-checked using the Alt key to highlight the pixels.
With yesterday's method, the one that stops at precisely 255, the values ​​in the text box were 220.

With the Alt key, the pixels light up at 220, then going back 1 point, they're at 221, the pixels turn off.

So the Alt key gives you 1 point, which I think is better for the highlights.

So much so that, although yesterday's result is excellent, the highlights are slightly high, still within the parameters, but they lose a little texture. Now I'll try again at 221.

[C.Oldendorf]

In the intermediate DNG used for interpolation we find:

Code: Select all

  | | 11) BlackLevel = 8943 8943 8943 (8943/1 8943/1 8943/1)
  | |     - Tag 0xc61a (24 bytes, rational64u[3])

That comes from the CR2 but is scaled by 4
(doubled once per bit step from 14 bit in our CR2 to 16 bit - spread out in the DNG)

Code: Select all

  | | | | AverageBlackLevel = 2047 2047 2047 2047
  | | | | - Tag 0x0114 (8 bytes, int16u[4])

The situation we see is that the black point correction is done but the data shifts to the left histogram wise leaving roughly 32 histogram values in 8 bit logic unpopulated at the right end. That is interesting, I'll need to look at what I do in MakeTiff in that regard and how to fix these edge cases. The libRAW interpolation branch scaled the data as it should.

[robyferrero]

I'm glad it's interesting.

If it helps, I can also attach the first one I showed in previous posts, the blurry one.

I'm not sure what you mean by saying this problem didn't occur with LibRAW.

In any case, the photo published in previous posts is interpolated with LibRAW, and the problem still occurred.