Improving the Mk. II scanner speed & resolution

This is the first followup to the Mk. II scanner article. In this article I'll cover two major improvements, one that massively improves scanning speed, and one that significantly improves the resolving power.

Hardware upgrades

I did some printing work, including a new set of film-plane holders with reduced backlight-spacing (the initial version had maximised the distance, but this proved unnecessary). Freeing up some height let me slip a 3 mm thick glass plate behind the Epson film holders, keeping it flat. Without this support the center dropped down by around 1 mm vs. the corners.

A new lens hood was printed with set-screw clamping to allow Z-offset adjustment (the original variants had fixed heights and were taped on). This lets me precisely control the clearance vs. the film holders.

The final print in this round was a slip-on "pocket" to fit a fiber media converter on the back of the top of the Y-axis mount. The 12 V from the scanning computer powers both the camera and the converter. While mainly done for personal æsthetic reasons, keeping the electrical link short should improve reliability. Further, anchoring the cabling near the camera prevents movement caused by tugging on wires.

While this converter is a 1000Base-T to 1000Base-X (fiber) converter I had sitting around, in future this will be replaced with a simpler "OEO" converter of the same size. Optical/electrical/optical converters are electrically trivial devices that simply connect SFP(+) modules' TX/RX data lines together, making them transparent (though the link-speed must match for both modules). This converter will make the networking part ready for 10 Gbit⁄s Ethernet in future.

Getting at the raw sensor data = speed

While trying to work out which parts of my signal chain were limiting factors, I noticed that the time taken to grab a single frame had grown over time as I added more options (like CCM's).

As it turns out the MDVision SDK does all processing in software, and calling the frame processing function (to convert the raw sensor data into 16-bit RGB values) took 3-500 ms per frame. Actually transferring the raw image took 50-150 ms (it jumps around a bit). This is huge since debayering and similar operations can be done later, sitting around waiting for that to happen for every frame is pointless. I also expect performance of debayering to be better when done on the merged HDR image instead of e.g. debayering partially saturated frames.

I had previously been attempting to switch the camera into raw mode to play around with other debayering algorithms but found this didn't work. At this time I realised that what I was giving to the processing function was literally just a pointer to raw sensor data.

I found an optimised 12-bit packed to 16-bit unpacked function on Stackoverflow and within a short while I was outputting the bayer image instead of an RGB image.

It took a few hours of rework to handle this change, but I was able to make it work. Mainly this meant the display thread now had to do both debayering and colour/sensor corrections that were previously done in the main image pipeline. I then had to add the same functions in the merging pipeline. I gained an appreciation for how expensive floating point matrix operations are on a CPU.

Debayering comparison

For preview I use the OpenCV debayering, which does seem ok but it only supports 8/16-bit data which is fine for raw sensor data, but not fine for my integrated HDR images. The loss of shadow detail would be significant when doing linear-light operations.

I decided to use the Colour-Science project's debayering implementation, which I suspect isn't enormously optimised for speed in the Python implementation.

I do wish the documentation had a bit more to say in why you'd pick a given algorithm, but the results are quite good. I went with the "DDFAPD" algorithm (Demosaicing With Directional Filtering and a posteriori Decision) since it seems to be most advanced one. I let it do the optional refining step as well.

It seems there's a small bug in how the algorithm is implemented where image edges are slightly glitched (containing very large or negative values), as noted in a Github issue. I work around this by simply clipping the image after processing. Cropping slightly would also work just fine.

The IMX183 seems to be a RGGB order sensor, since as far as I can see of the debayering code it does use R/G/B order. However, my realtime processing using OpenCV processing works correctly with BGGR order. I don't think there's a meaningful difference in processing of red vs. blue so this seems to be a case of flip until it looks right.

As should be clear the performance of the more advanced algorithm is far superior for fine details. Some rainbowing effects are observable but we can in fact see every line quite clearly which is rather impressive given how few pixels are involved, and the lines being quite diagonal.

Performance

Previously scanning a full roll took 4 hours if they were slides (the worst case). With these changes but no change to integration counts etc., scanning a 35 mm frame can be done in ~1.5 minutes and a full roll takes somewhere around 1-2 hours now. I consider this to be entirely acceptable.

The merging script that I previously optimised so nicely (15 seconds per colour frame down from 50) shot back up to 50 seconds again. The demosaicing is the majority of this runtime, with the colour correction matrix step as runner up.

I managed to claw it back down to ~25 seconds by multi-threading more—this time I spin up one thread per input subframe and do all the debayering, defect pixel management, flat field correction, gamma, inversion, sharpening etc. at once before starting the sequential-ish merging operations. The actual merging and blending is now only a fraction of the processing time, which is nice I guess.

For a 35 mm frame doing pre-processing separately essentially doubles performance since it's a 2x2 subframe assembly, so previously two threads were doing what 4 threads do now (on a 12-core CPU we're single-thread performance limited). There's no obvious way to further improve on this since the per-subframe image processing is quite sequential; subdividing each subframe is technically possible but seems likely to cause issues with debayering (the most expensive operation).

I expect finding a way to run these operations on a GPU is the way to go, at present I don't have any experience doing this. I can certainly appreciate why low-performance and hardware-accelerated debayering is common now.

Standard lab-scan comparison

Here's a few frames from a Kodak Funsaver disposable camera scanned by Oslo Foto (standard quality ~6 MP; I don't have any examples of higher quality scans) (not otherwise adjusted) vs. my rescan with processing. This was simply the first un-scanned roll sitting around.

The results of my scanner seem pretty good, though I still haven't really worked out the correct colour correction matrix/profile to use so my reds tend towards oversaturated.

Note that the Oslo Foto scans are barely compressed (~7 MB per 6 MP frame) but do list sRGB colour space and as such will be limited in their colour rendition.