Color recovery concept from the AXIOM Alpha and the basis for packing raw data inside an RGB image in the AXIOM Beta
For the AXIOM Alpha, we came up with a method to output a color image at 1080p over HDMI from the 4k raw sensor data. We recorded that video feed on an external recorder which defined container and maximum possible resolutions (at the time there were no external UHD capable recorders available yet). With the AXIOM Beta, we maintained this approach but thought about how we could push the boundaries. Currently, with the HDMI plugin module we are limited to 1080p60 as the highest possible throughput mode (It's a frequency limitation of the lanes coming from the Zynq FPGA. A future plugin module with another FPGA onboard featuring gigabit transceivers will be able to output UHD/UK signals directly.). So we thought about methods where we could pack more data into the existing modes and the result is that we can capture 2160p30 inside a 1080p60 (or in the future 2160p25 inside 1080p50 and 2160p24 inside 1080p48) video on an external recorder connected to an AXIOM Beta. So far this mode was called “experimental 4K RAW”, but since it’s getting less and less experimental and we now know it works (we shot the entire April Fools Joke video this way) we actually dropped the “experimental” in the name.
Uncompressed 4K raw still image captured with AXIOM Beta
First Generation Raw Mode
We applied a gamma curve to the 12-bit raw data and directly mapped the Bayer channels to RGB output: R, (G1+G2)*0.45, B. Colors were graded in post production anyway so we didn't care much about the color accuracy on set at the time. However, because of the slight misalignment between the 3 channels, these images contained more resolution information than was needed for 1080p, but less than for UHD. We tried to recover the raw channels without much success, so we ended up attempting to recover a UHD RGB image by solving a modified debayering problem: instead of inputs being [R, G1, G2, B], our inputs were [R, (G1+G2)*0.45, B]. While the results were far from great, this experiment showed where the encoder was struggling, so we knew what to change in the second version.
The main points were:
-
Row/column noise was not corrected, which caused the codec to struggle, introducing very obvious compression artifacts
- Horizontal resolution was affected because (a) we have averaged the two green channels, and (b) the recorder applied a 422 chroma subsampling, which affected the details in red and blue channels.
- Output was not covering the full 12-bit range (this was actually the least of our worries)
Second Generation Raw Mode
To solve the downsides from the previous generation, we have implemented the following changes:
- Recorded the image at double frame rate (1080p60), for the purpose of combining every two HDMI frames into a single UHD frame (3840x2160).
- To fix the resolution loss in the green channel, we sent the Bayer G1 channel on even frames, and G2 on odd frames.
- To fix the resolution loss in the red and blue channels, caused by 422 subsampling, we sent the Bayer R/B channels on the even frames from the HDMI stream, and a shifted version (by one pixel) on the odd frames.
- Subtracted the static components of the row/column noise. Ideally, we should have subtracted a complete dark frame, but this is not yet possible in the current FPGA firmware. Therefore, we have corrected the row/column offsets before sending the data to HDMI (because they were the main reason for the codec struggles), and subtracted the rest of the dark frame from the HDMI image in post-production.
- To minimize the image alteration caused by encoding the linear 12-bit data as 8-bit, we have designed an optimal LUT (similar to a log gamma), based on the noise profile of the image, that attempts to throw away only bits of noise, without altering the original signal, if possible.
- In a nutshell, we ended up sending (R,G1,B) on even HDMI frames, and (R’,G2,B’) on odd frames, after applying a LUT. Note: the apostrophe ‘ means “shifted by one pixel”.
The two resolution recovery tricks caused only a tiny (1-pixel) flicker in the preview image on the recorder, which was not noticeable at all in practice. The optimal LUT actually gave a usable preview image - not color correct, but sufficient for monitoring purposes, and containing enough information for recovering the original Bayer raw data almost completely.
In other words, not only have we successfully recovered the UHD 12-bit Bayer data from two 1080p 8-bit RGB frames with 4:2:2 chroma subsampling, but we also have the mathematical background for recording linear raw data at lower bit depths, with minimal loss. The optimal LUT for lowering the bit depth can be found from the noise profile of the input image - that is, from the SNR curve. For details on this process, please see our Log Curve Simulation Paper.
Second generation AXIOM Beta raw mode capturing 2160p raw data inside two 1080p frames (A & B) on external recorder
Conversion Process
There are 2 general steps in the process:
Lets say our AXIOM Beta shoot sample clip is called clip.mov. We recorded it on the Shogun so this clip is ProRes compressed.
Converts a video file recorded in AXIOM Beta raw mode to PGM image sequence (recovers the raw data from the compressed movie):
./hdmi4k clip.mov
hdmi4k source code on Github
Convert a PGM image sequence to a DNG sequence:
./raw2dng --fixrnt --black=120 clip*.pgm
raw2dng source code on Github
The --fixrnt does temporal row noise correction, and the black level adjustment is needed because the black level from this sensor appears to get lower than the level calibrated with dark frames, and the cause is not yet fully understood (for details see: this wiki page).
Both commands can also be combined to go directly from quicktime mov to DNG sequence:
./hdmi4k INPUT.MOV - | ./raw2dng --fixrnt --pgm --black=120 frame%05d.dng
Read detailed instructions on apertus° wiki
AXIOM Beta UHD raw image as result from combining A & B frames.
Notice the details in the magnified area are almost 100% recovered (compared to the uncompressed raw image).
Resolution Comparison
magnified Frame A, Frame B and the result of recovering raw data from frames A & B
AXIOM Beta UHD Sample Clips from April Fools Video
We know that making crazy jokes and producing weird video projects as April Fools Joke is not something one would expect from a camera manufacturer - but remember the AXIOM was born by filmmakers who were fed up with the tools they had to use and so they set out to create their own, so we are still artists at heart. And we were getting impatient already to try out the AXIOM Beta in a real production environment, so April 1st came round and there was our perfect opportunity. This was also the first footage captured outside our lab in UHD raw to test and improve the UHD workflow and gather experience using the Beta in the field in general. The conditions were quite extreme (night shots in the rain with high contrast scenes, green screen keying, etc.) so it was great to see how our image sensor handled these kind of situations and how our image processing can strive to make the most of it.
The following original unprocessed footage downloads provide short sample clips that you can convert to DNG sequences with the above software and scripts yourself. We also provide a DNG for every clip that we converted already for you.
What you see here is still work in progress and will improve and evolve in the future, for example the DNGs don't have white balance metadata and when converting the sample clips the first few frames will have visible row noise.
Calibrated darkframes for AXIOM Beta unit used to shoot the following clips (required for raw conversion):
download
Further Links
Want to participate?
Want to provide feedback or suggest improvements? Want to try your own approach and need help?
Get in touch via IRC or email.
