ColorPerfect 3, GFX 100S Pixel Shift DNG, PhotoLine and light source questions

[pkr1979]

Hi,

My setup: a Fujifilm GFX 100S on a copy stand with a Pentax 645 120mm macro, used for digitizing color and black-and-white negatives in 135, 120 (645 and 6x7) and 8x10" large format. I use PhotoLine as my editor and have an existing ColorPerfect license I am considering upgrading to ColorPerfect 3. Part of what I want to explore is Fujifilm Pixel Shift Multi-Shot: the camera records 16 RAW exposures which Fujifilm's Pixel Shift Combiner merges into a single 400 MP DNG file.

My priority is the highest possible image quality rather than the fastest or simplest workflow, so any advice on the recommended workflow would be much appreciated.

1. Fujifilm GFX 100S
I understand the GFX 100S is supported by PerfectRAW. Does this support also apply to the GFX 100S Pixel Shift Multi-Shot DNG files, or only to the normal RAF files?

2. Pixel Shift DNG
Can ColorPerfect 3 / PerfectRAW / MakeTiff process the 400 MP DNG files produced by Fujifilm Pixel Shift Combiner correctly, while preserving the full RGB information and resolution?

3. DNG support
Can ColorPerfect 3 work directly with DNG files, or is MakeTiff required to convert them to a linear TIFF first? If MakeTiff is required, is the Fujifilm Pixel Shift DNG format supported?

4. PhotoLine
I use PhotoLine and understand it is fully supported as a host for ColorPerfect 3. Can PhotoLine open and process the 400 MP Pixel Shift DNG files correctly, and can these files then be passed to ColorPerfect 3 without losing the relevant RAW/RGB information?

5. Stitching
I am also interested in combining several GFX captures of the same negative — for example two or more overlapping Pixel Shift captures of a larger negative — to obtain an extremely high-resolution reproduction. Would you recommend:

Pixel Shift DNG → stitching in PhotoLine → ColorPerfect 3, or
processing the individual captures in ColorPerfect first and stitching afterwards?

6. Scanned negatives
I also have negatives digitized with a conventional film scanner. Can ColorPerfect 3 be used with these scans as well, preferably from high-bit-depth TIFF files?

7. Light source
My copy stand uses a Kaiser Slimlite Plano LED panel, run at full brightness. Are ColorNeg's film calibrations valid with a white-LED light source, or do you recommend a specific illuminant for digital capture of color negatives? I am aware that white LEDs have a spiky spectrum compared to the illuminants traditionally used in scanners, and I would like to know whether this affects the accuracy of the built-in film characterizations — and if so, whether there is a recommended way to compensate.

Cheers
Peter

[robyferrero]

Hi Peter,
I can't give you all the answers, but you'll see that everything will be clear.

In the meantime, I can tell you what I know and what I believe.
First, my advice is to upgrade to CP3 just for the DC function, but not only that.
The DC function will work perfectly with digital camera negative reproductions; try it and believe it; there's a big difference.

CP3, like CP2, seems complex, perhaps even slow, but when you use it every day, gain experience, and learn what it's for and how each slider works, it immediately becomes easy and, above all, fast. I also prioritize image quality over speed, and with CP, I have both.

Regarding the workflow, what can I say? I could tell you I'll explain mine, but what's the point? Perhaps it would be helpful if you explained yours, or better yet, if you could explain the problem to us, if you run into any difficulties.
I don't think so, but if you're just starting out, feel free to ask. I'm not the one who can give you all the answers, but out of everyone here, you'll find the solution.

1) As you surely know, you need to feed your RAF file into MakeTiff, and the same goes for the DNG file, and I'd assume that Pixel Shift Multi-Shot or not makes no difference.

The same goes for your question number 2: MakeTiff converts the DNG to Linear Tiff.

In fact, if you need to remove noise or make various corrections before converting and processing in CP, you can, for example, generate a DNG file from DxO Pure Raw and then feed it into MakeTiff.

Therefore, your questions numbers 3 and 4 are the same as the previous answers.

5) I'm no expert, but I know it's best to stitch files before processing them, but processing can be more complicated than expected: it's not a problem with CP or other software, it's that the camera, for one reason or another, including the frequency of the LED light source, generates files that aren't perfectly even in exposure and color temperature. It seems to me that a very specific shutter speed needs to be identified to prevent the LED from flashing. I saw a video somewhere; it's in Italian, but I could find it if you need it.

6) As you may know, linear scanning is best. However, with CP2/3 you can get a decent result from anything

7) I think most people use an LED light source.

Of course, I can't tell you if you've worked extensively with CP2 or with other software, but if you're not very familiar with CP, I can tell you that you'll be amazed at the results you can get from your scans or digital camera reproductions. Obviously, the better these files are generated, the better the results will be.

Among other things, we'd also like to see the results.

[pkr1979]

Thanks!

I did not know this:
1) As you surely know, you need to feed your RAF file into MakeTiff, and the same goes for the DNG file, and I'd assume that Pixel Shift Multi-Shot or not makes no difference.
I am only now starting to digitize film with a camera (up until now Ive used scanners - and ColorPerfect).

Cheers
Peter

[C.Oldendorf]

Hi Peter,

first, thank you, Roberto, for contributing your practical experience with ColorPerfect 3 and DC mode. I will build on what you have already explained rather than repeat it.

In a nutshell, I expect Peter’s proposed setup to work. We have not yet tested the precise combination of a GFX 100S Pixel Shift DNG and PhotoLine, but since Peter has the necessary camera, we can test this together and establish the results properly.

How Fujifilm Pixel Shift Multi-Shot works

For readers unfamiliar with the process, the GFX 100S uses its in-body image stabilization system to move the sensor between exposures. One Pixel Shift sequence records 16 separate RAF files, with the sensor displaced by precisely controlled whole-pixel and half-pixel increments.

The first part of the process uses four captures displaced by one whole pixel. Because the sensor has a Bayer colour-filter array, each position would ordinarily measure only red, green or blue, and the two missing colour components would have to be interpolated from neighbouring pixels. Moving the sensor by one pixel between the four exposures places the different colour filters over the same image positions. Fujifilm can therefore derive actual red, green and blue measurements for every position at the sensor’s native resolution.

Fujifilm Pixel Shift Combiner calls the corresponding 100-megapixel output Accurate Colors. It retains the nominal resolution of the sensor but eliminates the need for conventional Bayer interpolation. For that reason, I expect this output to be superior to a normally captured single RAF, which still has to be interpolated.

For the second output mode, Fujifilm repeats this complete-colour acquisition at four half-pixel-displaced positions. The four resulting full-colour grids are then combined to double the horizontal and vertical pixel dimensions, producing a 400-megapixel DNG. Fujifilm calls this mode High Resolution + Accurate Colors.

Thus, the 100-megapixel output derives its advantage from obtaining actual colour information at every pixel position, while the 400-megapixel output additionally uses the half-pixel displacements to increase the nominal spatial resolution by a factor of four. Neither mode is simply an unusually elaborate version of ordinary Bayer interpolation.

I expect both outputs to work with PerfectRAW. If Fujifilm’s half-pixel reconstruction has any relevant implications, they would be expected in the 400-megapixel result, not in the basic 100-megapixel Accurate Colors result. This is presently an expectation rather than the result of a test on the files in question.

DNG, MakeTiff and ColorPerfect

PerfectRAW’s GFX 100S support is not limited to ordinary RAF files. The files produced by Fujifilm Pixel Shift Combiner are linear DNGs, and I do not expect a significant difference in colour processing between a regular RAF and either of the combined DNG outputs.

MakeTiff can process the 400-megapixel Fujifilm DNG as long as the DNG remains below the current 2 GB file-size threshold. Fujifilm specifies an approximate size of 1.5 GB, so this should present no difficulty.

The size limit is an implementation detail concerning the signed or unsigned treatment of the offsets used in the TIFF-based DNG format. Larger combined files, such as those produced by Hasselblad Phocus, require a custom implementation using unsigned offsets throughout. I have completed such an implementation but have not yet released it. Consequently, even files between 2 and 4 GB are not a fundamental obstacle.

ColorPerfect itself cannot process a DNG directly. It is a Photoshop-compatible plug-in and requires unadulterated pixel data supplied by its host. The workflow therefore remains the one Roberto has already described:

DNG → MakeTiff → linear TIFF → ColorPerfect

PhotoLine

PhotoLine provides a dedicated function under File → ColorPerfect Import. This is the route to try with the Fujifilm DNG. I do not currently have one of these Pixel Shift DNGs here, so I cannot report a firsthand result for this exact file and hardware combination. Peter can readily establish this with one of his files, and we can assist if anything unexpected occurs.

Stitching several captures

The best order should also be established in a practical test. My initial expectation is that negative conversion before stitching will be the more stable approach.

The individual linear TIFFs would first be processed in ColorPerfect using exactly the same settings, employing ColorPerfect’s exact carryover mechanism. The resulting positive images should provide the stitching software with stronger contrast edges and more readily identifiable detail than the comparatively low-contrast negative originals.

I would therefore begin with the following order:

Individual DNGs → MakeTiff → identical ColorPerfect processing → stitching of the positive images

Conventional scanner files

There is little to add to Roberto’s answer here. High-bit-depth TIFF files from conventional film scanners are, of course, supported. Processing such scans has always been one of ColorPerfect’s principal applications.

LED illumination and the film characterizations

ColorNeg’s built-in film characterizations do not characterize a particular scanner, camera or illuminant. They describe the gamma portion of the film type concerned, essentially the straight slopes of its characteristic curves.

For digital-camera reproduction, the relevant camera-side characterizations are those supplied by PerfectRAW and also used by ColorNeg DC mode.

Any reasonably daylight-like light source is suitable. The discontinuous spectrum of a white LED does not present the same colour-accuracy problem in this application that it can when photographing a real-world scene containing millions of differently coloured and potentially metameric surfaces.

When reproducing a colour negative, the camera is observing the three dye systems of that film at varying densities and therefore varying brightness. Provided that the LED source differentiates those three dyes suitably, the irregularities in its spectrum do not affect colour accuracy in the suggested manner. No special illuminant compensation is required, and a daylight-like LED panel is a sensible choice.

ColorNeg DC has also been tested under much more extreme illumination. A colour enlarger head can, for example, be adjusted with its colour filters so that the combination of the illuminant and the sensor’s intrinsic channel sensitivities largely eliminates the visible orange mask during capture. This gives the blue channel a massive increase in signal and therefore potentially substantially more usable detail than a standard capture.

The cost of this approach is not necessarily poorer source data but a considerably more demanding mathematical situation during processing. Even these extreme cases hold up reasonably well, but the practical recommendation remains to use a reasonably daylight-like source.

Testing the complete combination

Peter is in an excellent position to provide a controlled comparison. An ordinary RAF, the 100-megapixel Accurate Colors DNG and the 400-megapixel High Resolution + Accurate Colors DNG of the same negative would allow us to compare the normal MakeTiff-interpolated capture with both Pixel Shift outputs.

This would verify the complete MakeTiff and PhotoLine workflow and show directly how the 100-megapixel full-colour result and the additional 400-megapixel reconstruction perform in actual negative reproduction. I expect all three routes to work, so let us test them and document what we find.

[pkr1979]

Thanks a lot for this. I can provide files. I will try to get it done this weekend but might need the next one. I got a busy couple of weeks.

Also - I think PhotoLine stitches. So maybe stitching in PhotoLine before opening the image in CP3 might be a good option as well... I don't know. Just thought so as it is happening in the same program.

[robyferrero]

Stitching several captures

The best order should also be established in a practical test. My initial expectation is that negative conversion before stitching will be the more stable approach.

The individual linear TIFFs would first be processed in ColorPerfect using exactly the same settings, employing ColorPerfect’s exact carryover mechanism. The resulting positive images should provide the stitching software with stronger contrast edges and more readily identifiable detail than the comparatively low-contrast negative originals.

I would therefore begin with the following order:

Individual DNGs → MakeTiff → identical ColorPerfect processing → stitching of the positive images

This is an interesting point.
Personally, I never question Christoph’s suggestions because who better than him can give us the correct answer, the best one, or the most likely hypothesis? So I think that's the case here as well.

However, I believe it’s not easy to capture a frame with, say, four different sections and then stitch them together so that they’re all perfectly identical in exposure and color; I think there are very specific techniques to follow to overcome the problem of the LED’s discontinuous spectrum—is that possible?

For example, it seems to me that even a slight difference in color can make one section appear slightly brighter or darker than another; these aren’t significant differences, but they’re still capable of slightly altering the consistency of the final result.
The risk is that very faint “bands” will form between one section and another, as they may have a brightness and/or hue that differs slightly from the others.
Consequently, even the exact same CP processing might not yield perfectly identical results, which—when sewn together—lead to the problem just described.

In short, it may be necessary to perform comprehensive tests, such as:

A) Individual DNG files → MakeTIFF → identical ColorPerfect processing → merging of the positive images.

B) Individual DNGs → merging the images (is it possible to obtain a stitched file in DNG format?) → MakeTiff → ColorPerfect.

C) Individual DNGs → MakeTiff → stitching the Linear TIFF → ColorPerfect.

This last example, which works well in Photoshop, is easy to test quickly: take any Linear TIFF file, duplicate it, crop 1/3 off the first one and 1/3 off the second, then stitch them together and process the result with ColorPerfect. It turns out perfect. I don’t know if this will work in all cases, but in the ones I’ve tried, it turned out perfect.

[pkr1979]

Hey - here are the sample files.

[C.Oldendorf]

Let’s look at the black-and-white positive reproduction first. All versions work well.

Simply use ColorPos mode and set G/L to L for linear input. Alternatively, configure the Options to always assume linear input. Set Saturation to 0. If you want to influence how the scanner’s red, green, and blue channels contribute to the result, use Tonal to blend them to your liking.

[Full image link - opens in new tab]

In principle, the stitched capture is processed in exactly the same way. For the second half, use an exact carry-over of all settings from the first half. Make sure that G/L is set to L again.

That gets us here:

[Full image link - opens in new tab]

There is a visible line in the image because the stitching does not work properly, even when image deformation is permitted. Why? The two captures do not register accurately, as we can see along the edges after stitching.

The example below was stitched with only translation and rotation permitted:

reg_mismatch.jpg (41.5 KiB) Viewed 211 times

[Full image link - opens in new tab]

The reproduction setup therefore needs improvement. Editing this out would be difficult:

reg_mismatch_image.jpg (37.82 KiB) Viewed 210 times

[Full image link - opens in new tab]

Quite apart from the reproduction problem, the photograph itself provides a lovely illustration of the power of the Scheimpflug principle. The plane of sharp focus does not have to remain vertical: the upper and lower parts of the foreground tree are out of focus while its middle is sharply rendered. The nearer houses then drift slightly out of focus, yet buildings farther away become sharp again. The plane of focus cuts diagonally through the landscape as the terrain descends towards the city.

[C.Oldendorf]

Moving on to the color negative, it can be converted using ColorNeg DC. The version shown here was made from the computed four-shot DNG.

[Full image link - opens in new tab]

Pictorially, we see much the same effect: a diagonally tilted plane of focus created through Scheimpflug movements, combined with late light and very strong vignetting. The image seems to invite a panoramic final format, although I have deliberately left it uncropped here.

The more important point concerns the reproduction photography—specifically exposure level and the sensor’s ADC bit depth.

We have already discussed why getting the exposure right for negative reproduction is both crucial and somewhat counter-intuitive. The camera’s JPEG histogram and clipping warning do not reliably show what is happening in the linear sensor data. For a fixed reproduction setup, the film base provides the brightest possible part of the negative. Exposure should therefore be set so that the film base uses virtually the full sensor range without actually clipping, preferably retaining about one third of a stop of safety margin. ColorPerfect’s Headroom indication or a genuine linear histogram can be used to establish this.

This matters because linear sensor values are not distributed evenly across the available stops. The brightest stop receives half of all available ADC values, the next stop one quarter, and so forth. Dense parts of the negative that are recorded far down the linear range consequently receive far fewer tonal values and sit closer to the sensor noise. Maximising the normal reproduction exposure therefore preserves considerably more usable information than the camera preview might suggest.

This particular negative reveals an additional problem. Even when the ordinary capture is exposed as fully as possible, the densest areas transmit so little blue light that the blue channel still contains insufficient detail:

blue_channel_vs_red_channel.jpg (270.3 KiB) Viewed 205 times

[Full image link - opens in new tab]

Highlight compression would expose this weakness mercilessly, so I did not use it here. Instead, I shaped the image with Filmic Relight and Zones.

For exceptionally dense color negatives, however, there is a better capture-side solution: first make the normal exposure at the maximum level permitted by the film base, then make a second exposure at +2 EV. The second capture is not intended to replace the first one. Its purpose is solely to give the severely underexposed blue channel four times as much exposure. Clipping in the red and green channels is irrelevant because those channels will be discarded.

Keep this second capture linear. In Curves, set input 255 to output 63, scaling its values back to approximately one quarter and thus compensating for the additional two stops. Then transplant only its blue channel into the normally exposed image.

The resulting blue channel retains the correct relative level but was recorded much higher within the sensor’s linear range, with better tonal differentiation and greater separation from the noise floor. The normally exposed red and green channels remain untouched. In effect, this is a channel-specific exposure extension for negatives whose density and color filtration prevent all three channels from being captured optimally in a single exposure.

[C.Oldendorf]

The black-and-white negative does not present the same channel-specific exposure problem. The dense snow areas transmit less light, of course, but the negative has no orange mask or unequal dye densities starving one sensor channel in particular. All three channels record essentially the same monochrome information, and the best channel—or a suitable Tonal blend—can be used.

The 100% view shows that even within the densest parts of the negative there is ample differentiation. The crystalline structure along the edge of the snow remains clearly resolved, with no shortage of usable tonal information:

[Full image link - opens in new tab]

The complete conversion therefore requires neither an additional +2 EV capture nor any channel transplantation:

[Full image link - opens in new tab]

This is the important contrast with the color negative shown above. With black-and-white film, exposing the reproduction as fully as the film base permits is sufficient. With a very dense color negative, however, the normal exposure can use the sensor’s overall range optimally while the blue channel in the densest areas still falls too low in that range. That is the special case in which the additional +2 EV exposure and corrected blue-channel transplant can recover information that the ordinary capture does not record adequately.

[C.Oldendorf]

Slide film causes none of these complications. A well-exposed reproduction photograph can simply be processed in PerfectRAW mode as though it were a photograph of the original scene:

[Full image link - opens in new tab]

That is possible because slide film was designed to be the final image, both in its color filtration and in its tonal rendition. There is no inversion to perform, no orange mask to compensate for and no severely disadvantaged blue channel to reconstruct. Provided that no substantial color correction with CC is required, the result should be straightforward.

More extensive color corrections can become complicated, however. The characteristic curves of slide film do not retain the same slope throughout their entire range: the brightest couple of stops behave differently from the middle and darker parts of the image. A global correction can therefore have different effects in different tonal regions, making selections or otherwise localized corrections necessary.

This varying slope is essentially part of the analogue solution to having no explicit black point. The film’s gamma compresses the deepest tones towards full density and thus zero intensity range in the final positive image—even when it is viewed by projection—rather than relying on a digital black-point operation afterwards.

[pkr1979]

Thanks for this Christoph,

I tried the other way around - I made the maketiff - then stiched in Hugin - then opened in PhotoLine and ColorPerfect. It worked quite well, except stiching uniform areas doesn't work that good (I tried the E6 one). Stiching in PhotoLine did not really work at all for me. If I were to use PTGui I would need the pro version and that is just way to expensive. This isnt a CP-issue though - which is working with all scenarios, but a stitching issue. Which I am not completely sure how to resolve - it seems to be more hazzle then I expected it to be.

Cheers
Peter

[pkr1979]

PS I saw just now that you tested other files too. My previous comment was just a response to your comment on the black and white reversal - I will read your other comments with keen interest too

[C.Oldendorf]

Yes, given that you had provided four different kinds of material, I wanted to say something about each of them.

The fundamental registration challenge is keeping the sensor plane and the reproduction-medium plane exactly parallel. Even a very slight camera tilt—which can easily occur with a camera mounted somewhat freely on a copy stand—will cause the two captures to differ geometrically. Simple translation and rotation will then no longer be sufficient to register them perfectly.

Essentially, we want to reproduce what a view camera already allows us to do very precisely: keep two planes parallel and locked in place. I have long thought that a monorail camera could be useful as the basis of such a reproduction setup, although I have never actually built one. A 4×5 monorail would probably be too small for convenient work, so something based on 5×7 might be more practical.

There are simpler possible constructions as well. If the camera is mounted to a rigid board and supported by a precisely cut tube or similar spacer of the required length, the camera and lens can be held perpendicular to the reproduction plane. The camera then remains completely fixed, and only the original needs to move underneath it. The easiest arrangement might be to place the original on a light panel that can be repositioned accurately without changing its height or orientation.

For trying the alignment in Photoshop, place the two captures in one document as separate layers, select both layers in the Layers panel, and choose Edit > Auto-Align Layers. This command is certainly present in CS5 and possibly earlier.

Photoshop offers several projection modes. Collage is probably the most appropriate starting point here: it aligns the layers without distorting their shapes, allowing the necessary repositioning, rotation and uniform scaling. With a reproduction setup that maintains a constant camera distance and does not change its tilt, Collage should work splendidly.

I ultimately used Auto for the black-and-white positive. This permits Photoshop to choose a more extensive transformation, including some stretching or skewing, and it got us considerably farther with the imperfectly registered captures. The same procedure should work in principle with the other types of film as well. Naturally, exposure and illumination must remain absolutely constant between the two captures if they are to blend without a visible transition.

There is one important complication if you want to stitch the captures before ColorPerfect. The MakeTiff data is linear, but it is tagged with the profile intended for the later ColorPerfect output. At this intermediate stage, the profile’s tone reproduction curve therefore does not describe the actual linear image data.

If you understand precisely what this means, you can make a linear-gamma version of your intended output profile in Advanced Color Options and assign that profile temporarily. Photoshop will then understand that the image data is linear and can preserve the correct linear relationships while aligning and blending the layers.

After alignment and after merging the layers to your satisfaction, assign the original profile expected for the ColorPerfect workflow again—for example, Adobe RGB (1998), if that is your chosen output space—and then proceed with ColorPerfect.

These operations must be profile assignments only. There must be no color-space conversion at any point before ColorPerfect. A conversion changes the pixel values themselves and therefore destroys the required relationship between the linear data and the profile ColorPerfect expects, particularly in ColorNeg mode, as we saw in this recent discussion.

If that distinction between assigning and converting profiles—or the reason for temporarily using a linear-gamma profile—is not entirely clear, it is much safer to process both halves separately with exactly identical ColorPerfect settings and stitch them afterwards.

So yes, this is a stitching problem rather than a ColorPerfect problem. Nevertheless, it is very much an adjacent problem and therefore of interest to me and, I expect, to many people here. That is basically what I hoped we could do on this forum: talk not only about ColorPerfect itself, but also about photography and all the related fields of our beautiful art.