Cinelerra based Workflow

Introduction

The research presented here aims at establishing a digital cinema editing workflow made exclusively under Linux machines. The steps described below start at the point in which we already have a sequence, or many sequences, of digital negative (DNG) files. In the diagram, they correspond to the stages immediately after the third grey box. Everything described from there on has been tested and is documented here.

Even though it is the aim of this documentation to gather information related to the whole process – which includes capturing and monitoring the recording, transferring the data to the computer, playing the JP4 files and finally converting them to DNG sequences - the previous parts are can be retraced by putting together information that can be found at Apertus’ site or that is spread throughout Elphel’s Wiki. We can consider that, therefore, as a second step towards our objective.

This research first tests which format is best to be used as proxy. It takes into consideration that editors will need to do many test renders during editing and that a fine photographic adjustment of the images will be done only at the final stages (those proxies, then, must be easily replaceable), in which the workflow is divided between the photographer, the audio technician and the final retouches by the editor.

Finally, this page is currently also mantained at the author's blog, where it should be translated to Portuguese.

Preparations for editing

Files to be used as proxies

We could render the DNG files generated by movie2dng using ufraw-batch. At this moment of the workflow, we are interested in generating proxies files - that is, light files that have three characteristics: they have to be able to used for editing; they have to present a good preview of the final video without compromising too much of the quality; and they have to register fast rendering times in our NLE (be it Cinelerra or Blender) so that we can do preview-renders of our edited video quickly and leave high CPU demands for post-editing.

To achieve so, however, we have to test which type of file would best fit into all these requirements. Would the best format be TIF of JPG? For the test below, I used 4 sample DNGs generated by Elphel downloaded from Apertus Project's website. I copied them and pasted them into the same folder so that I would have 360 frames - or a good preview of what to expect from 15 seconds of a CIMAX recording (2592x1120) at 24fps. Note that the frames I used were even larger than the CIMAX format.

Test 1

Process 360 DNG frames (occupying 3,4GB of disk space), or 15s of RAW video footage

Command line used:

ufraw-batch --conf=apertus_teste.ufraw *.dng

Results: For what we can see from this first test, four formats have the potential to be used as proxies: JPEG 50% PPG, JPEG 50% Bilinear, TIF Uncompressed Bilinear and TIF Uncompressed PPG. The first two have the advantage of occupying very low disk space if compared to the third and fourth ones (44MB~ x 5,1GB). They can be a very interesting solution, especially for larger projects. But if we consider the workflow as a whole, the TIF formats should make out life easier at post-production. The problem with this test is that the command line above processes only one image at a time. With some research, I came across a very simple software called parallel, that can be easily compiled (don't use the pre-packaged versions, they are too old) and will help us to use all the cores of a multi-threaded processor, in my case, the Intel i7 860. Dividing the work between the cores of the processor dramatically reduced the time in my tests - generally, it took him half the time to complete the task; in some cases, it took him one third of the time.

Test 2

Process 360 DNG frames (occupying 3,4GB of disk space), or 15s of RAW video footage - using multi-threaded processing

Command line used:

ls *.dng | parallel -j +0 ufraw-batch --conf=apertus02.ufraw --silent {}

Results: As it turns out, it seems that the best formats to be used as proxy are the JPEG 50% Bilinear and the JPEG 50% PPG. The observation about disk space occupied by both (see previous results) is still pertinent and reducing further the quality of the JPEGs (below 50%) may even fasten the overall conversion, but that must be tested in the timeline of the NLE, during a real editing project. The JPEG formats also benefited most from the multi-threaded task.

Diskspace worries

For the TIF formats, we must consider the enormous amount of disk space occupied by them. The table below is just a rough preview. I take into consideration only the DNGs converted by movie2dng and their processed TIF counterparts, by ufraw-batch. You should have in mind that there are still the original Elphel's JP4 files, (many) Cinelerra preview renders you should make along the way, the temporary files you should use as the project goes through the whole process, original audio, audio for post-production and the final movie render.

Rendering tests using the proxy files

It is now time to check how these image sequences will behave in our NLE. My initial intent is to use Cinelerra as editor and Blender for effects, such as titles or post-production. So I imported the files generated by the tests above into a timeline, using CIMAX standard as reference (2592x1120 at 24fps). Note that for this test, I use only the video stream, since it's too soon to preview which will be the best workflow for audio.

Test 3

Render 360 frames, or 15s of 2592x1120 video footage at 24fps from Cinelerra's Timeline

Results: The JPEG 50% Bilinear and JPEG 50% PPG are the fastest, both rendering at similar speeds. The difference in time when rendering these formats with the JPEG Photo codec at 50% or 100% is almost irrelevant, but the space in disk occupied by them should weight in favour of JPEG Photo at 50%'s side. It is worthy noticing the behaviour of JPEG brought to Cinelerra using AHD. It takes about 4x to render when compared to its cousins - if rendered with JPEG Photo at 100%, it also takes way more space in disk. Working with TIF here serves for us to have an idea of the necessary time to render the final version of the video, that should be brought here using TIF Uncompressed AHD and rendered either as a TIF Uncompressed Sequence (to be encoded by MEncoder for standard outputs) or as JPEG Photo at 100% with PCM audio in a MOV container.

Overall conclusion from the tests

There are two main conclusions to be drawn from these tests. The first one is that the JPEG 50% Bilinear and JPEG 50% PPG are the best ones to be used as proxies. They are the fastest to be processed, both during ufraw's batch conversion and cinelerra's render: they take 7x real-time to be processed at the first step and only 2x real-time at the second one. They also occupy minimum space in disk, and can be easily previewed by MPlayer at anytime during the workflow.

But there is a major drawback in using the JPEG formats. If we combine them with img2list, we'll have a hard time replacing the JPEGs at Cinelerra's timeline with the final TIFs due to how img2list and Cinelerra work together (Cinelerra's XML don't point to the images, but to img2list's generated list, which can't be changed without harming Cinelerra's interpretation of it). That should leave you two options. You can invert the workflow and do the photographic treatment before editing. That can be done for some movies; for others, it will be unthinkable. Or, more reasonably, you could replace the JPEG image blocks manually in Cinelerra's timeline for the TIF ones. That can be less work than it seems, but you'd have to have the very final cut of the movie at the moment of replacement, since further editing - even a minor tweak - will become quite hard. In other words, you'd be fronzen there.

The second conclusion is that, even though the TIF Uncompressed PPG and the TIF Uncompressed Bilinear will occupy way more space than the JPEGs and will take longer to be processed both at ufraw's batch and cinelerra's render phases, they may have advantages if you consider the workflow as a whole. Firstly, they will take 12x real-time at the first step and 7x real-time at render, should you combine them with JPEG Photo 50% for render previews. That is slow. However, that might be compensated at post-production. When you do the photographic treatment in UFRaw and generate new TIFs, you'll be able to simpy replace the TIFs you had used for the new ones in the folder. Cinelerra will read the alterings just fine, even though its XML points to the img2list's lists. The lists, in their turn, are already pointing to TIF files (this won't work for JPEG, though. You won't be able to use, for example, JPEG at 100% and replace the JPEG proxies at the folder - Cinelerra will break if you do that).

This means that you will still have your image sequences behaving like movie blocks at the timeline, which is crucial. In case you must change that single frame that went unnoticed or change the duration or order of anything, that should be quite smooth and effortless. You will also lose MPlayer's ability of previewing a sequence of images as a file, which can be vey handy - but since the TIFs can be replaced at the folder and instantly read by Cinelerra, you might be able to check them directly at the timeline.

In both cases, though, you will have to be very organized with your files and UFRaw's configuration files. Which way is best? Consider your project; consider the gear you have and judge for yourself.

Editing

To edit the image sequences, we should use a tool called img2list, developed by Claudio 'Malefico'. If we simply import these frames into Cinelerra, they will be treated as single images by the software. Well, that's what they actually are, but img2list will help Cinelerra read the frames as a 'sequence of images' (that is, a movie), which is exactly what we want to do. Now, they will behave exactly as movie blocks in the timeline. You will be able to split them, to stretch or shrink them exactly as if they were, for example, a DV file. As an aditional comment, img2list will work only if the image sequence is named in a certain pattern, which happens to be compatible with the pattern used by movie2dng.

Editing in Cinelerra is quite well known and very well documented, so I will skip the introductory steps here. Rendering the video in Cinelerra to preview the final result should take into consideration the tests presented above, so you should probably want to render the video in a MOV container, using JPEG Photo at 50% or 100% as codec settings.

Post-production

Image treatment should be done using the original DNG files, for the simple reason that they are RAW. Both Cinelerra and Blender are able to open DNG files from cameras, such as Pentax's DNGs. But it seems that only Cinelerra will open Elphel's converted DNGs without having to recompile the software. To work with DNGs in Cinelerra will be extremely time consuming, though. Minor tweaks in colour or slightly altering contrast will take an enormously long time to be previewed, transforming a delicate process into nightmarish hell (that's the main reason why we transformed the original DNG into JPEG proxies in the first place).

A reasonable option would be to make Blender read Elphel's DNG-converted files and use its compositing nodes tool to do the colour correction. That has yet to be tested. Also, we would have to establish a communication between Cinelerra's EDL (a XML file) and Blender, so that we could import our EDL in Blender.

UFRaw, however, has the right tools and immediate preview. Its main problem is that it lacks the time-lapse factor (that is, you can only view a single, still, frame). That can arranged in terms, using MPlayer to preview the processed sequence (see below), but it should present difficulties in scenes that have moving cameras or strong changes in contrast. The following will consider a workflow using UFRaw.

First of all, we need to know which DNG files we'll be working with. It would make no sense to go through DNGs that belong to recorded sequences we didn't use in the final editing cut. The information we need is inside Cinelerra's EDL, which is a XML file. By going through this file, you can know precisely which frames have to go through post-production. An example of the section we need inside Cinelerra's XML is (click on image to enlarge):

This excerpt show two very small blocks of video in a single track, called "Video 1". The first one uses 7 frames, there is a 3-frames space between the blocks and then there is a 6-frames video block. Visually, it would look like this in your timeline:

Now we must translate that information into human-readable terms. It must be simple to understand. We can make a script using the long command line below. For file "cinelerra.xml" as input, it will give us a file called "List_of_DNGs_for_post_production.txt", which is a text file you can print or read in the computer, the way you feel more comfortable with. The line:

grep "EDIT STARTSOURCE=" cinelerra_outra_pasta.xml | cut -d">" -f1,2 | cut -d"=" -f1,2,4,5 > temp_readableXML.txt && sed -ie 's/<EDIT STARTSOURCE="/- frames /g' temp_readableXML.txt && sed -ie 's/" CHANNEL="/ to /g' temp_readableXML.txt && sed -ie 's/"><FILE SRC=/ File: /g' temp_readableXML.txt && grep File temp_readableXML.txt > temp_readableXML2.txt && gawk '{ $8 = $5 + $3; $9 = $3+1 ;print $6,$7,$1,$2,$9,$4,$8 }' temp_readableXML2.txt > List_of_DNGs_for_post_production.txt && rm temp_readableXML*.txt temp_readableXML.txte

Will give us this more reassuring output in the text file:

File: "/home/livre/Desktop/testes_e_exemplos_elphel/dngs/img2list/lista" - frames 1 to 7 File: "/home/livre/Desktop/testes_e_exemplos_elphel/dngs/img2list/lista" - frames 8 to 13

Now it is clear which DNGs we should use. In this case, I'd just go through my file "lista" (which is a img2list file I had created previously for editing) and see which DNGs I'll have to reprocess in UFRaw and ufraw-batch. The easiest way to do that would be to copy those files to a temporary folder, treat them and check them out directly at Cinelerra's timeline or with MPlayer:

Command line used:

mplayer "mf://*.jpg" -mf fps=24:type=jpg -fs -vf dsize=2592:1120

When you're satisfied with the results, copy the resulting files (probably Uncompressed AHD TIFs) and paste them into the original DNG's folder. If you have used Uncompressed TIFs as proxies, you will be prompted to replace the TIFs in that folder. Do it, replace them. And you're done.

Now, when you open your Cinelerra project again (that cinelerra.xml file, in our example), the program will read your new TIFs instead of the old ones and you're ready to mix the other final sources (audio and lettering) for a final render.

Lettering and other effects should be done in Blender. Depending on Blender's behaviour, we can use proxies to do it and export the result using an alpha channel, so that it can be brought into Cinelerra's timeline for the final render. In case Blender is able to read our original files, we can do the final render inside it. In both scenarios, a Render Farm can be built to help the CPU efforts.

Research continues...