Raw photos converting

Here are some notes about imaging with Sony NEX-5 camera and converting its raw files into ready images. Majority of these tips is applicable to other cameras as well.

Manufacturer of the camera provides software for RAW converting: Image Data Converter and Imaging Edge. It's reasonable to assume that these tools are adjusted for Sony cameras and know all their characteristics. But they do not give linear brightness, which is essential in any serious work, their algorithm is unknown and, moreover, the first of them is obviously half-baked. In addition, such programs create beauty at the cost of truth. More mighty and transparent tools are also available in Internet, but they need characteristics of the particular camera. The same applies to conversion with self-made scripts, but they are, at least, fully controllable and changeable.

This page concerns converting raw files with the help of scripts for Matlab. Such approach gives access to linear brightness values, and, hence, is compatible with operations like flat-field correction.

Main steps of the conversion:

• • decoding raw (.ARW) files into 16-bit .tiff without any changes of brightness;

  • reading them in Matlab, where all the following work will go on;

  • accounting for darkness and saturation levels;

  • camera calibration;

  • white balance correction;

  • demosaicing;

  • transformation of colors into standard color space;

  • gamma correction.

Noteworthy issues with Sony Image Data Converter

As of January 2018, the program has some notable bugs:

• • when converting a single file into tif, setting "16 bit" does not work: image becomes 8-bit in any case. To obtain 16-bit tifs, you should use batch conversion;

  • sometimes output image contains thin and hardly notable line looking like some juncture, running vertically across the image's center;

  • noise reduction is implemented even when turned off. So, very small details are always spoiled (which would not be the case if noise is removed by frame averaging without noise reduction).

In response to my report of these bugs, I was told that the program is no more supported by the manufacturer. Its successor, Imaging Edge, works better, but only on 64-bit systems.

Besides the bugs, following details are notable:

• • output of the program (as well as jpg output of the camera) is cropped by 8 pixels on each side and has size 4592x3056 px, while original size (preserved by conversion tools like dcraw) is 4608x3072 px;

  • output of the program with default settings and jpg output of the camera are not identical in brightness, although very similar.

Primary decoding of raw images

Work with raw images begins with their converting to some readable format without changing any brightness values. Such converting can be made with dcraw by Dave Coffin (.exe file can be downloaded here). In Windows 7, you can:

• • put the program (.exe file) into the directory with raw files;

  • holding "Shift" key, right-click any empty place in the directory and select "Open command window here";

  • paste in the command window the string

 dcraw-9.27-ms-32-bit.exe -v -t 0 -T -j -4 -D *.ARW (in 32-bit Windows)

or

.\dcraw-9.27-ms-64-bit.exe -v -t 0 -T -j -4 -D *.ARW (in 64-bit windows)

where *.ARW means processing of all ARW files in the folder (for processing a single file, replace these symbols with its name);

• • press "Enter".

In the mentioned string, command "-v" makes the program to display information about the work, "-t 0" ensures no rotation (ignoring of the camera's gravity sensor data), "-T" is for writing tiff files, "-j" for "not stretching or rotating raw pixels", "-4" for linear 16-bit output without white balance correction and "-D" for writing brightness values without scaling ("totally raw"). Full list of incantations can be found here.

After this, you will have tiff files with brightness values identical to original ones. They are still not viewable, because they are neither brightness-adjusted nor demosaiced, but now suitable for processing by tools like Matlab.

Short aside: complete conversion with dcraw

Dcraw can also convert raw files into ready viewable images (the same commands as above, but without "-4" and "-D"). For better results, you should also add some other things (otherwise the program uses proper settings, which can be nonoptimal). These things include, but are not limited to:

• • command -w, which tells the program to use white balance coefficients from the image's metadata;

  • file of the dark frame (example of the command: -K darkframe.PGM with uppercase "K") or a single number — mean brightness of such frame (-k 128 with lowercase "k");

  • saturation brightness (-S 4090).

Default dcraw's values for dark and saturation brightness of Sony NEX-5 are 116 and 4095 respectively. The above-mentioned values 128 and 4090 are actual values for my camera of this model (my report of this bug had no consequences). It's noteworthy that 4095 is 212-1, and 128 is 27. So, images of this camera are actually almost 12-bit.

So, the command for complete conversion should look like

dcraw-9.27-ms-32-bit.exe -v -t 0 -T -j -w -k 128 -S 4090 *.ARW

Default values of the parameters, used by dcraw, come from the program's code, which contains data for a huge number of cameras. For Sony NEX-5, it has even 2 sets of parameters (of which the first is used):

    { "Sony NEX-5", 116, 0, /* DJC */

       { 6807,-1350,-342,-4216,11649,2567,-1089,2001,6420 } },

...

    { "Sony NEX-5", 0, 0, /* Adobe */

       { 6549,-1550,-436,-4880,12435,2753,-854,1868,6976 } },

"116" stands for darkness level, and numbers "6807...6420" are coefficients of the XYZ-to-camera color transformation matrix.

Dark and saturation brightnesses

Precious work with dark and bias frames can be a whole science, but for images of bright objects by Sony NEX-5 it is sufficient to subtract a single number: 128.1. This parameter practically does not depend on exposition time (in all the available range 1/4000 — 30 s), ISO (in all the available range 200—12800) and temperature (in the range of its growing during taking a set of several dozen images). During my experiments with different exposition times and ISO values, mean brightness of dark frames varied between 127.6 and 128.3. The difference constitutes 0.02% of the total interval between dark and saturation values.

Saturation value on images by Sony NEX-5, measured on my specimen, is 4090. For my Canon EOS 30D, dark and saturation brightnesses constitute 128 and 4071.

Camera calibration

Adobe Digital Negative (DNG) specification, which gives a detailed algorithm for processing of raw images, describes a correction for individual camera's color sensitivity. This correction is specified by a "camera calibration matrix", which "transforms reference camera native space values to individual camera native space values" (since we need reverse transformation, the matrix must be inversed). Where to find "camera calibration matrix" and how to interpolate it from 2 given variants, see below. For my Sony NEX-5, this matrix (identical in both variants) is:

    0.9415     0       0

    0          1       0

    0          0       1.0057

In my case such calibration is harmful: if it is applied, white objects look pinkish.

White balance

White balance correction should be done before demosaicing (a good comparison of "before" and "after" can be seen on a page by Guillermo Luijk). Coefficients for the correction, used by camera and image-processing software, are saved in the file's metadata and can be extracted with dcraw using command

dcraw-9.27-ms-32-bit.exe -i -v *.ARW.

The program will display text like:

    Filename: DSC08397.ARW

    Timestamp: Sun Dec 17 10:55:32 2017

    Camera: Sony NEX-5

    ISO speed: 200

    Shutter: 1/5.0 sec

    Aperture: f/5.6

    Focal length: 16.0 mm

    Embedded ICC profile: no

    Number of raw images: 1

    Thumb size: 1616 x 1080

    Full size: 4608 x 3072

    Image size: 4608 x 3072

    Output size: 4608 x 3072

    Raw colors: 3

    Filter pattern: RG/GB

    Daylight multipliers: 2.106416 0.958156 1.256952

    Camera multipliers: 2560.000000 1024.000000 1276.000000 1024.000000

The needed coefficients are "Camera multipliers" in the last string. They should be normalized to 1 in minimum (becoming 2.500 1.000 1.246 1.000 in the given example) and multiplied by brightness in R, G1, B and G2 pixels respectively.

Demosaicing

The most primitive methods of demosaicing process R, G and B color layers separately (e.g., red values on non-red pixel positions are obtained from interpolation of neighbouring red pixels regardless of green and blue ones). These methods give worser images then more elaborated ones, but they are worth mentioning because of one advantage: the result is resistant to chromatic aberration. However, this advantage can be negligible.

Here you can find a Matlab code for demosaicing images using 2 methods of this type: bilinear interpolation (which gives rather blurry images) and triangulation-based cubic interpolation (with significantly sharper results).

Matlab has also a special function for demosaicing: demosaic. It uses a more clever method, which takes into account the correlation among color channels: gradient-corrected linear interpolation by Malvar et al., 2004. I didn't find code of this function in Matlab and, for test purposes, wrote de novo a code which implements the same algorithm. Usually Matlab's demosaic function works identically with this code, but sometimes it gives slightly other results for images with black edges (if you blacken some part of the input image, it affects demosaic's output on all the rest of image, not only along the borders of black area; evidently, this is wrong behaviour). In addition, Matlab's function requires uint8 or uint16 input (in version R2013a), while self-made code for Malvar's algorithm accepts double input.

Dcraw offers several methods of demosaicing, of which the default is adaptive homogeneity-directed (AHD) interpolation. It gives a characteristic artefact pattern of short vertical and horizontal lines and is regarded as a "slow and inferior" method.

The best (and the slowest) demosaicing method I have tested is LPA-ICI based interpolation ("local polynomial approximation - intersection of confidence intervals") by LASIP project. Its result is similar to result of Malvar's algorithm, but without typical artefacts of the latter (too dark and colorful pixels on the borders of dark areas). To use this method, you should unpack ZIP package from the previous link and add there the function function_stdEst2D.m from here. After adding the folder from the package to Matlab search path, demosaicing can be performed with

     demosaiced_image = function_cfai_lpaici(mosaiced_image, CFA_phaseshift);

where CFA_phaseshift is 2 in the case of usual Bayer filter. Both sizes of the input image must be even. Due to huge memory consumption, images should be processed part by part.

Almost the same quality can be obtained with LSLCD-NE demosaicing by Jeon, Dubois, 2013. It is designed for noisy images and combines demosaicing and denoising (which introduces some inevitable blurring). The degree of denoising is controllable by multiplicating the variable esigma by some coefficient (1 = standart denoising, 0 = no denoising [as in the slideshow below]). For using this method with usual Bayer filter,  1-pixel-wide column must be added at the left side of the input image (and removed after the processing). Maximal sharpness is obtained with parameters CAorEA='CA', KorL='L'. Links to Matlab codes for this one and some other demosaicing algorithms can be found here.

Color conversion

For correct displaying on monitors, camera sensor's colors must be converted into standard sRGB color space. This is done in following way:

     Rnew = a11×Rold + a12×Gold + a13×Bold,

     Gnew = a21×Rold + a22×Gold + a23×Bold,

     Bnew = a31×Rold + a32×Gold + a33×Bold,

or, in matrix notation,

The matrix a11...a33 (hereafter cam2rgb following the designations of Sumner, 2014) depends on the camera model; relatively small dependence on color of illumination also exists.

Detailed algorithm of the color transformation is described in Adobe Digital Negative (DNG) specification and implemented in a script for Matlab on this page. To calculate cam2rgb, you need some parameters of the camera and one parameter (white balance data) of the photo. For a large number of cameras, necessary parameters can be obtained with the help of Adobe DNG Converter. This program converts raw images into DNG files, and then you can read the parameters from DNG's Exif with a program like RawDigger.

Since some of the needed parameters depend on the color of illumination, they are given in two variants: "1" and "2", which correspond to "Calibration Illuminant 1" and "Calibration Illuminant 2". In the case of Sony NEX-5, these illuminants are "Standard Light A" (with color temperature 2856 K) and "D65" (6504 K). The DNG specification requires finding correlated color temperature of illumination for each photo (on the basis of white balance data) and interpolating each parameter between variants "1" and "2", using inverse values of the temperatures for weighting (if color temperature of illumination for a given photo falls outside the range of color temperatures of "Calibration Illuminants", the specification prescribes taking of closest variant instead of extrapolation). But dependence of color matrices upon illumination isn't very strong, so it's possible to simply take one of the variants instead of interpolation. Variant "1" must be better for reddish illumination, and "2" — for whitish.

The DNG specification describes constructing of matrices, which include corrections for white balance and, if needed, for the camera's individuality. But it is better to apply these corrections to images beforehand (without including them into cam2rgb matrix), because:

• • white balancing should be done before demosaicing, and color conversion — after it;

  • if you have "Forward Matrix", constructing of cam2rgb is more simple and transparent in the case of corrected images;

  • matrices, which are not mixed with white balance correction, are more easy to check for their own correctness: they must convert white into white ([1; 1; 1] into [1; 1; 1]), so, sum of each row must be equal to 1;

  • such matrices are comparable with each other.

So, all following text describes cam2rgb matrices for images with mentioned corrections already applied.

The matrix cam2rgb can be obtained from data in DNG's Exif in 2 ways:

• • from "Forward Matrices" (recommended by DNG specification). The "Forward Matrices" 1 and 2 transform white balanced colors of reference camera (not individual camera) into XYZ colors with D50 reference white;

  • from "Color Matrices" (for the cases when "Forward Matrices" are absent). The "Color Matrices" transform XYZ values (not tied to D50) to native space values of reference camera.

Calculating cam2rgb from "Forward Matrices"

If the DNG's Exif contains "Forward Matrix 1" and "Forward Matrix 2", then

   cam2rgb = xyz2rgb * cam2xyz,

where:

• • cam2xyz is interpolated from "Forward Matrix 1" and "Forward Matrix 2" or simply taken as one of them;

  • xyz2rgb is a constant matrix for transformation of D50-based XYZ colors into standard D65-based sRGB colors. This matrix (3.1338561...1.4052427) can be found on Bruce Lindbloom's site.

Make sure that the sum of each row of cam2rgb equals to 1.

Calculating cam2rgb from "Color Matrices"

It is a more complicated way; for details, look code on this page (namely, calculating of cam2rgb_m2_clean). This method (but for images, not corrected for white balance, and only with conversion into D50-based XYZ colors) is described on page 81 of DNG specification. It's noteworthy that in the case of some input data you can obtain cam2rgb with sums of rows equal to each other, but different from 1. Such matrix will transform grey into gray of other brightness and must be normalized by division of each row by its sum.

Much simpler, but apparently well-working, way of calculating cam2rgb from "Color Matrix 2" (which is called ACam←XYZ or xyz2cam there) is described and Matlab-coded by Rob Sumner (pages 9-11). "Color Matrix 1" is not appropriate here due to large difference of the corresponding color temperature with 6504 K of sRGB-related illuminant D65. Matrices, obtained from "Color Matrix 2" with method from DNG specification and with Sumner's method, are somewhat different and give noticeably (but not much) different images.

"Color Matrices" are also used by above-mentioned program dcraw (ignoring their dependence on illumination color). Its code contains such matrices from Adobe DNG Converter and other sources for a large number of cameras. For Sony NEX-5, it contains two matrices:

   6807,-1350,-342,-4216,11649,2567,-1089,2001,6420   ("DJC")  and

   6549,-1550,-436,-4880,12435,2753,-854,1868,6976    ("Adobe"),

which means:

   "DJC":                      "Adobe":

   0.6807 -0.1350 -0.0342      0.6549 -0.1550 -0.0436

  -0.4216  1.1649  0.2567     -0.4880  1.2435  0.2753

  -0.1089  0.2001  0.6420     -0.0854  0.1868  0.6976

The "Adobe" matrix is "Color Matrix 2" from Adobe DNG Converter's output, and origin of the "DJC" matrix is unknown to me. Dcraw itself uses the "DJC" matrix. I tested both and found that difference between processed images is small and "Adobe" variant seems to be slightly better. However, this doesn't matter for the mentioned camera: cam2rgb can be calculated in a better way, because "Forward Matrices" are available.

Ready matrices for Sony NEX-5

Here are cam2rgb matrices calculated for Sony NEX-5 using 2 methods from DNG specification (in these cases the matrices depend on correlated color temperature of the illuminant) and Sumner's method. The first two methods are implemented in this script.

It is seen that in the case of illumination by "Calibration Illuminants", matrices calculated from "Forward Matrices" and "Color Matrices" are almost identical, but in the case of other illuminants they become somewhat different (but still produce almost identical images). Images, processed with the uppermost and lowermost matrices, differ significantly, but remain more or less realistic even in the case of using inappropriate matrix.

Gamma correction

To be properly displayed on monitors, linear brightness values must be transformed according to some tonal (gamma) curve. Four curves are worth mentioning:

• • the most widespread standard: an equation from sRGB specification. In Matlab language:

output = ...

(12.92 *  input                    ) .* (input <= 0.0031308) +...

(1.055 * (input .^ (1/2.4)) - 0.055) .* (input >  0.0031308);

This curve is very close to

output = input.^(1/2.2);

• • An older television standard: ITU Radiocommunication Sector Recommendation BT.709, the default curve of dcraw. In Matlab language:

output = ...

(4.500 *  input                 ) .* (input <  0.018) +...

(1.099 * (input .^ 0.45) - 0.099) .* (input >= 0.018);

• • Response curve of your camera. It can be obtained from comparison of raw images (after dark subtracting and color conversion) with ready output of the camera or accompanying raw-processing software. More clever methods, which don't even need raw images, also exist.

Such curves can heavily differ from sRGB curve: manufacturers distort them in order to make photos more attractive. Sony is not an exception.

Camera response can be viewed as consecutive applying of some "pre-sRGB curve" and sRGB curve. The overall curve and "pre-sRGB curve" applied by Image Data Converter with default settings to raw photos of Sony NEX-5 is shown on the first picture below. Curve applied by the camera itself is apparently the same.

The "pre-sRGB curve" used by Image Data Converter can be approximated with:

output = @(input) [...

0.96170                             - ...

1.32165 * sin(input*pi   - 1.37600) - ...

2.50258 * sin(input*pi*2 + 0.00918) + ...

2.49073 * sin(input*pi*3 - 1.75675) + ...

1.65385 * sin(input*pi*4 - 0.35140) + ...

0.76636 * sin(input*pi*5 + 1.07499) + ...

0.23129 * sin(input*pi*6 + 2.51870) + ...

0.03690 * sin(input*pi*7 + 3.98248)];

So, applying of this function and then sRGB curve emulates gamma encoding by Sony NEX-5. Reverse transformation - linearizing the photos - can be made with an ugly, but working formula:

sony_to_linear = @(x)[...

(x<=0.6736)            .*(0.0017 + 0.0924*x +0.1771*x.^2 -0.2551*x.^3 +0.4215*x.^4) + ...

(x> 0.6736).*(x<0.9438).*(0.2158 + 0.02346*tan(x*1.55963) -0.6850*x +0.7885*x.^2)   + ...

(x>=0.9438)            .*(316.5033 - 948.54374*x + 940.03554*x.^2 -307.01163*x.^3)];

• • Curve for precise displaying on your monitor. Can be derived from response curve of the latter, obtained from photographing of the monitor displaying artificial grey images with brightness varying from 0 to 255. Corrections for nonuniform brightness distribution on the monitor and for instability of exposition time should be applied. The first correction is derived from a photo of the monitor displaying an image of uniform brightness, and the second one - from a zone of constant brightness added to all the artificial images.

Response curve of an LCD monitor NEC MultiSync 90GX2 Pro, obtained in this way, is shown on the second picture. Colors are R, G and B of the camera used for photometry, not the monitor. It is seen that the curve is linear in the right half. Observed saturation of the green channel occurred in the camera, not the monitor. It must be noted that the curve depends on the monitor settings (during the measurements, brightness and contrast were set to 35%).

Tonal curve for correct displaying on this monitor, obtained as fitting of inverse function of R part of the monitor's response curve, is following:

output = @(input) [...

(input <  0.11384) .* (1.101738*input + 0.044088 * log(1 + input * 311.7734)) + ...

(input >= 0.11384) .* (0.705971*input - 0.01885./(0.06047 + input) + 0.01885/0.06047)];

Exposition time

How precisely does the camera measure time? Does it implement the settings correctly? Precise knowledge of exposition time is essential, in particular, for determination of characteristic curve of the camera.

Exposition time can be measured from photographs of moving objects with known velocity:

     time = (value of blurring) / (velocity of the object).

The appropriate object — a small movie like the shown one — is simple to create in Matlab:

For better diplaying, frame rate of the video should coincide with refresh rate of the monitor. That's why the movie should't be in the form of GIF animation: time of frame displaying in GIF is multiple of 0.01 s and can not correspond to most monitor values like 1/75 or 1/60 s. For better precision of measurements, the moving object must move to 1 pixel every frame, and the frame rate must be as big as possible.

If you are sure that your camera has linear response to illumination, you can also calculate ratios of exposition time of two RAW photos from their brightness.

Results