In the realm of digital imaging, there are many questions artists have regarding the nature of digital imaging. How do the scenes we look at get captured and transformed into meaningful colored and black and white images? How do the complexities of post production pipelines interact with data? How much should an artist need to know about color and color spaces to elevate their work? Color is called a psychophysical phenomena; a portion of color relies on the physics outside of our eyes and a portion rests on physiological and psychological processes within our bodies. Color does not exist beyond the biological entity of the human body, and as such, there are a tremendous number of contextual factors that determine how one reads visual stimulation. The following addresses the strictly numerical and data based aspects of color, and does not attempt to discuss the nature of the many color appearance phenomena that form a large part of visual perception. This post will attempt to offer some insight on a fascinating and complex subject, while ideally stimulating more interest in the area of color and imaging. No dresses will be discussed.

Biological Cameras

Our eyes are biological cameras in a way; they take a complex environment and render it into meaningful and useful data signals for our brains to process. While some might suggest that our eyes render reality, nothing could be further from the truth. Eyes are biased organic modules that transform complex data. That complexly transformed data is then, in turn, further transformed as it travels through even more complex processes that end up at the idea of an image in our minds.

In their most basic form, our organic sensors control the range of light that enters them, renders the light as biological data signals, and finally our minds weave the results into visual perception. The representation we see relies as much on learned and emergent phenomena as it does on the physics beyond them. When we look at visual stimuli, we read them.

Electronic Cameras

Electronic cameras are relatively simple creatures when we compare them to their biological counterparts. There are some unique differences of course. In particular, they are largely agnostic energy collectors, caring nothing of the reading and interpretation of the data they are attempting to measure. The sensor itself gathers units of light into discrete positional wells, also known as photosites or sensels.Interestingly though, cameras are incapable of seeing color, as they lack the complex contextual biological machine that crafts color. As biologist Timothy H. Goldsmith said, “Color is not actually a property of light or of objects that reflect light. It is a sensation that arises within the brain.” At their most basic level, camera sensors distill down wavelengths and intensities of light through a few different mechanisms. Contemporary sensors filter the photosites to deliver quantitative values of the scene. For a color sensor, each small photosite has a unique filter atop of it, typically divided across three general colors. With a filter, the photosite filters out an irregular wavelength of light, while permitting a mixture of other wavelengths through to a lower level data counter. Common contemporary sensors are organized around a tri-color light model; three unique filters divide the light into three unique value sets across the entire sensor. Using these three axes of data, a volume, or gamut, of light can be expressed.

Measurements to Meaning

While our eyes filter a scene's light into a distinct range of light and our minds process the information, their electronic siblings lack the latter half of that process. Instead, the data is gathered to the best the hardware can render, and it is up to a human to calculate and craft meaningful color from the data.

In a typical color CMOS sensor, the data is stored in a Bayer array as it is gathered from each of the photosites. This is a unique checkerboard pattern that alternates back and forth between pairs of photosite filters. While the actual sensor orientations may vary, the general idea is to alternate between green and red photosites for one row, and blue and green photosites for the next. This twin pairing of rows then alternate all the way down to the last row on the sensor.

The green filtered photosites comprise over half of the entire sensor, while blue and red fill out the remaining half of the photosites between them, or a quarter of the total count each. Why does a Bayer sensor privilege the green photosites over red and blue? Because the human perceptual system tends to be biased toward intensity over color, or luminance over chrominance. Although red, green, and blue might seem to be equal citizens in the land of color, greenish and yellowish hues live much higher up the luminance scale. Of the many receptors in an average eye, the most common are sensitive to this range of information, and as such digital sensors leverage a green photosite bias.Similar to a sensor, when we speak of a traditional pixel, we generally are referring to a tri-colored set of data at an individual position in an image or display. As you may likely have noticed, a Bayer sensor has only a single color at each position. How then do we end up making a pixel from only a single type of photosite at a singular position? The imaging engineer must take the unique checkerboard data values from the sensor and distill them down, via a technique called debayering, into the more traditional pixel RGB triplet.

Raw to RGB

There is often much mysticism around the term raw. Raw data implies that the data is a lowest-level representation, but what most artists end up seeing is never such. As explored with the creation of the AXIOM Alpha, the data must be massaged and sculpted to make it useful. Low level hardware complexities and issues often result in a quantity of the data range being unsuitable for imaging. This requires clever developers to cut, clip, bend, and otherwise craft the irregular information into useful data. With commercially available DSLRs and motion picture cameras, it is this post-massaged data that gets labelled as “raw”, despite being thoroughly “cooked”. But even after the engineer has massaged the data into useful tricolored counts of data, what exactly do we have?

Counts to Color

When the Commission Internationale de l'Éclairage, or CIE, conducted their canonical research in 1931, what was created was a unique map for referencing human color perception. This map resulted in a means to reference color in an absolute sense, color appearance phenomena aside. With the 1931 experiment, it finally became possible to discuss color using precise numerical values in relation to a standard observer, using an absolute color model with an implied absolute color space. Whenever an artist encounters terms or values around color, it can be said with certainty that they are referring to values in relation to the XYZ color model defined in the historic 1931 experiment.

Sadly, unbeknownst to far too many artists, the RGB color model is a relative color model. Specifically, it is a model that communicates values of intensity with no inherent communication of color. When an artist is presented with RGB data, she knows nothing of the actual colors, or chromaticities, implied by each of the arbitrarily labelled red, green, and blue channels. Identical sets of RGB values such as 0.4, 0.5, and 0.6 have entirely different meanings depending on the systems they reference. In order to create meaning from RGB intensity values, more context must be delivered along with it. When the raw RGB values are finally delivered at the end of a complex engineering path, they represent quantities of filtered light. What is unknown however, is the color of the intensities recorded. The filters on the original photosites of the sensor are not filtering narrow wavelengths of light per se, but rather a complex and baked weave of wavelengths that are intertwined with the other sensor filters. This makes the transformation of values into meaningful XYZ positions even more tricky than it might first appear. To mitigate this dilemma, charts or lightboxes containing carefully measured CIE 1931 XYZ values are photographed. After capture, the data off of the Bayer sensor is massaged into singular pixel values per coordinate. As a final step, the unreferenced series of data values in the image are compared and analyzed against the known data values from the original sample, and a dataset of relations is created. The result is a method to decode the raw and arbitrary sensor data into meaningful color in relation to the 1931 CIE standard observer model. Only after this process do the terms red, green, and blue take on any semblance of meaning; they have ceased being arbitrary data values and now reference absolute CIE 1931 XYZ values in relation to the values originally photographed.

Lost in (Color) Space

As we can see, every unique combination of sensors, filters, hardware, and other such dark alchemy forms a complex system. For every complex system, we end up with a particular and unique transformation of data into meaningful CIE 1931 XYZ color values. As such, we could say that for each camera and set of contexts there is a unique volume of color, or color space, that is represented. Each unique set of contexts require a unique transformation pattern to convert the data into a meaningful output context. While it might be logical to try and have a single color space to “rule all color spaces”, the reality is that our color representations are human-created models, they respond in unique ways when we apply our human-created mathematics. That is, for every color space an artist will find themselves facing potentially different results; different RGB color spaces will yield different CIE 1931 results. Further, at varying points in an artist’s imaging pipeline, an artist might find herself with unique imaging needs which might require a unique set of color space transformations. At the most simple transformation, an artist might wish to view her work on a display. To properly display the color from our camera described above, we would also need to know the characteristics of the display she is attempting to display it on. That is, just as the camera has a unique set of hardware that determines its color space, so too does the display, and as such we could say that the display has a unique color space, or characterization, of its response. When we know the camera’s color space and the display’s, we are able to merge the two datasets into a Rosetta stone and map the camera’s data values to the display’s.

The End of the Beginning

The above post dives, with any luck, just-deeply-enough for a majority of artists that are interested in the imaging side of cameras and color spaces. If there is enough interest, subsequent posts might dive more deeply into some of the tangled netting of handling color and imaging within a pipeline. AXIOM has a long way to go to becoming a full-fledged tool for imagers, but even the greatest camera is useless if artists aren’t aware of the foundation that rests at the base of their work. Color forms the foundation of all digital visual art, and as such, it’s a critical sphere of knowledge to grasp. Hopefully this post managed to kindle enough interest in your creative head to ask more questions. If it did, please ask and we will do our best to deliver answers.