For the AXIOM Alpha, we came up with a method to output a color image at 1080p over HDMI from the 4k raw sensor data. We recorded that video feed on an external recorder which defined container and maximum possible resolutions (at the time there were no external UHD capable recorders available yet). With the AXIOM Beta, we maintained this approach but thought about how we could push the boundaries. Currently, with the HDMI plugin module we are limited to 1080p60 as the highest possible throughput mode (It's a frequency limitation of the lanes coming from the Zynq FPGA. A future plugin module with another FPGA onboard featuring gigabit transceivers will be able to output UHD/UK signals directly.). So we thought about methods where we could pack more data into the existing modes and the result is that we can capture 2160p30 inside a 1080p60 (or in the future 2160p25 inside 1080p50 and 2160p24 inside 1080p48) video on an external recorder connected to an AXIOM Beta. So far this mode was called “experimental 4K RAW”, but since it’s getting less and less experimental and we now know it works (we shot the entire April Fools Joke video this way) we actually dropped the “experimental” in the name.
First Generation Raw Mode
We applied a gamma curve to the 12-bit raw data and directly mapped the Bayer channels to RGB output: R, (G1+G2)*0.45, B. Colors were graded in post production anyway so we didn't care much about the color accuracy on set at the time. However, because of the slight misalignment between the 3 channels, these images contained more resolution information than was needed for 1080p, but less than for UHD. We tried to recover the raw channels without much success, so we ended up attempting to recover a UHD RGB image by solving a modified debayering problem: instead of inputs being [R, G1, G2, B], our inputs were [R, (G1+G2)*0.45, B]. While the results were far from great, this experiment showed where the encoder was struggling, so we knew what to change in the second version.
The main points were:
- Row/column noise was not corrected, which caused the codec to struggle, introducing very obvious compression artifacts
- Horizontal resolution was affected because (a) we have averaged the two green channels, and (b) the recorder applied a 422 chroma subsampling, which affected the details in red and blue channels.
- Output was not covering the full 12-bit range (this was actually the least of our worries)
Second Generation Raw ModeTo solve the downsides from the previous generation, we have implemented the following changes:
- Recorded the image at double frame rate (1080p60), for the purpose of combining every two HDMI frames into a single UHD frame (3840x2160).
- To fix the resolution loss in the green channel, we sent the Bayer G1 channel on even frames, and G2 on odd frames.
- To fix the resolution loss in the red and blue channels, caused by 422 subsampling, we sent the Bayer R/B channels on the even frames from the HDMI stream, and a shifted version (by one pixel) on the odd frames.
- Subtracted the static components of the row/column noise. Ideally, we should have subtracted a complete dark frame, but this is not yet possible in the current FPGA firmware. Therefore, we have corrected the row/column offsets before sending the data to HDMI (because they were the main reason for the codec struggles), and subtracted the rest of the dark frame from the HDMI image in post-production.
- To minimize the image alteration caused by encoding the linear 12-bit data as 8-bit, we have designed an optimal LUT (similar to a log gamma), based on the noise profile of the image, that attempts to throw away only bits of noise, without altering the original signal, if possible.
- In a nutshell, we ended up sending (R,G1,B) on even HDMI frames, and (R’,G2,B’) on odd frames, after applying a LUT. Note: the apostrophe ‘ means “shifted by one pixel”.
The two resolution recovery tricks caused only a tiny (1-pixel) flicker in the preview image on the recorder, which was not noticeable at all in practice. The optimal LUT actually gave a usable preview image - not color correct, but sufficient for monitoring purposes, and containing enough information for recovering the original Bayer raw data almost completely. In other words, not only have we successfully recovered the UHD 12-bit Bayer data from two 1080p 8-bit RGB frames with 4:2:2 chroma subsampling, but we also have the mathematical background for recording linear raw data at lower bit depths, with minimal loss. The optimal LUT for lowering the bit depth can be found from the noise profile of the input image - that is, from the SNR curve. For details on this process, please see our Log Curve Simulation Paper.
Lets say our AXIOM Beta shoot sample clip is called clip.mov. We recorded it on the Shogun so this clip is ProRes compressed.
Converts a video file recorded in AXIOM Beta raw mode to PGM image sequence (recovers the raw data from the compressed movie):
./hdmi4k clip.movhdmi4k source code on Github
Convert a PGM image sequence to a DNG sequence:
./raw2dng --fixrnt --black=120 clip*.pgmraw2dng source code on Github
Both commands can also be combined to go directly from quicktime mov to DNG sequence:
./hdmi4k INPUT.MOV - | ./raw2dng --fixrnt --pgm --black=120 frame%05d.dng
Read detailed instructions on apertus° wiki
AXIOM Beta UHD Sample Clips from April Fools Video
We know that making crazy jokes and producing weird video projects as April Fools Joke is not something one would expect from a camera manufacturer - but remember the AXIOM was born by filmmakers who were fed up with the tools they had to use and so they set out to create their own, so we are still artists at heart. And we were getting impatient already to try out the AXIOM Beta in a real production environment, so April 1st came round and there was our perfect opportunity. This was also the first footage captured outside our lab in UHD raw to test and improve the UHD workflow and gather experience using the Beta in the field in general. The conditions were quite extreme (night shots in the rain with high contrast scenes, green screen keying, etc.) so it was great to see how our image sensor handled these kind of situations and how our image processing can strive to make the most of it.
What you see here is still work in progress and will improve and evolve in the future, for example the DNGs don't have white balance metadata and when converting the sample clips the first few frames will have visible row noise.
- Newsletter editions - Previous, monthly, html newsletters archived. Subscribe here to receive yours via email.
- Shop links, prices and availability
- Camera structure - A complete guide to the camera's specifications and components.
- AXIOM Beta sample videos - Uncompressed .mov .mp4 and .dng files in 720p, 1080p and 4K are archived here if you'd like to play with them.
- Pre-order notification - Receive an email when a more user-friendly version of the camera is ready to ship.
- GitHub Repository
- Join the Team
- Social accounts - Links to YouTube etc.
- AXIOM Project Background - How apertus° and the AXIOM project got started.