Basic Raster Operations

Raster Operations

JOSE LOPEZ-COLLADO

June 2015

This page shows some basic aspect of raster operations on asc files. An asc file is a file written in an ESRI format, made of six header lines and a matrix of values containing cell values, the file is a text file, thus it can be processed with any text procesor.

The tutorial first open (imports) an asc file and removes the header lines so to handle only the data matrix. Then, there is some computations on the matrix to show the information the file contains: its dimensions, a map, histogram and basic statistics. it is assumed that the file is in the working directory, otherwise the full directory path has to be included.

Next, there are a few user-defined functions to process the data matrix. The first function standardize the values between zero and one, though Mathematica has a Rescale function, it should be modified to handle the null values in the asc file, which usually are defined by the -9999 value.

Another function is to binarize the data: 1 if values are within min and max values, 0 otherwise. Again, it is necessary to test for null values. Mathematica has a built-in function for images: Binarize[].

The previous functions apply directly to the data matrix, so it is necessary to make them Listable, that is, they can operate on the matrices directly.

A third function linearizes the data using a linear transformation, in the example, it uses the coefficients that model the development of Thrips palmi (McDonald et al 1999), obtained through linear regression. This function is an example of using Map to apply a function to the matrix.

The last function transform the data to probability values; first, an empirical distribution is generated, then a function to translate the empirical values to probability values is presented, using the built-in CDF (Cumulative Density Function). This transformation flattens the original distribution to range between 0 and 1 at similar intervals.

Notice:

The example file is in the File Box page

(*==================== HERE STARTS THE CODE =============*)

SetDirectory[NotebookDirectory[]];

(* Load ascii file as matrix, that is table, drop the first six lines or header *)

ver1 = Drop[Import["asc1fin.asc", "Table"], 6];

(* get the dimension of the data matrix *)

Dimensions[ver1]

{641, 610}

(* Take a look at the map *)

ArrayPlot[ver1, PlotRange -> {Full, Full, {0, Automatic}}, ColorFunction -> "TemperatureMap", Mesh -> {12, 8}]

(*Select valid values from matrix, usually the asc file contains -9999 as NULL value, you have to check which is the NULL value opening the file with a text editor like NotePad++ *)

v1 = Select[Flatten[ver1], # > -9999 &];

(*view matrix data as a histogram*)

Histogram[v1, BaseStyle -> {FontSize -> 14, FontFamily -> "Helvetica"}]

(*Get the basic statistics. *)

bstats = {Mean[v1], Min[v1], Max[v1], Variance[v1], Dimensions[v1]}

{23.8542, -0.7, 26.7, 9.34174, {88004}}

(*============= some user-defined functions ========================*)

(*

stdMap normalize data between 0 and 1. Mathematica has a built-in function Rescale, but does not take into account NULL values.

binMap transform the date to 0 and 1, 1 if cell values are within a min and max values.

lineart is a linear transformation. Here, it is defined to show using the built-in function Map and apply the linear function to the data, in the previous cases, the functions are set with the attribute Listable so it can be applied to matrices directly

*)

stdMap[xmap_, minx_, maxx_, nullv_] := If[xmap == nullv, nullv, If[(xmap <= maxx) && (xmap >= minx), (xmap - minx)/(maxx - minx), 0]];

SetAttributes[stdMap, Listable];

binMap[xmap_, minx_, maxx_, nullv_] := If[xmap == nullv, nullv, If[(xmap <= maxx) && (xmap >= minx), 1, 0]];

SetAttributes[binMap, Listable];

lineart[x_, b0_, b1_, tmin_, nullv_] := If[x == nullv, nullv, If[nullv < x <= tmin, 0, b0 + b1 x]];

(* Now transform the data between zero and one: *)

std1 = stdMap[ver1, -0.7, 26.7, -9999];

Histogram[Select[Flatten[std1], # > -9999 &], BaseStyle -> {FontSize -> 14, FontFamily -> "Helvetica"}]

(*

apply the linear transformation. Because the data is a two-dimension matrix, when using Map it has to set the level to two: {2} as the last argument. Coefficients -0.05 and 0.0005 and 10 were obtained by linear regression and published by McDonald et al. 1999

*)

devThripsPalmi = Map[lineart[#, -0.05, 0.005, 10, -9999] &, ver1, {2}];

v2 = Select[Flatten[devThripsPalmi], # > -9999 &];

Histogram[v2, BaseStyle -> {FontSize -> 14, FontFamily -> "Helvetica"}]

(*Generate the raster maps*)

f1 = ArrayPlot[std1, PlotRange -> {Full, Full, {0, Automatic}}, ColorFunction -> "Rainbow", Mesh -> {12, 8}];

f2 = ArrayPlot[binMap[ver1, 15, 25, -9999], PlotRange -> {Full, Full, {0, Automatic}}, ColorRules -> {-9999 -> White, 0 -> Gray, 1 -> Blue}];

f3 = ArrayPlot[devThripsPalmi, PlotRange -> {Full, Full, {0, Automatic}}, ColorFunction -> "SolarColors", Mesh -> {10, 8}];

(*

Compose the maps in a row

the first map is the normalized between zero and one, the second is binarized, the third is a linear transformation

*)

gr = GraphicsRow[{f1, f2, f3}, ImageSize -> 650]

(*

To further process the data, it can be exported as a text file. An option is to save the file as *.dat file, then, open it with a text processor, add the header lines with modifications for null values if necessary and save it with the extension asc

Export["myStdFile.dat",std1]

*)

(* Generate an empirical distribution *)

edist = EmpiricalDistribution[v1];

(* map the empirical distribution to the cumulative density function *)

feCDF[x_, nullv_, ed_] := If[x == nullv, nullv, CDF[ed, x]];

(* Apply to original values *)

edMap = Map[feCDF[#, -9999, edist] &, ver1, {2}];

(* Display the transformed data *)

ArrayPlot[edMap, ColorFunction -> "Rainbow",

PlotRange -> {Full, Full, {0, 1}}, ImageSize -> 350]

(*Show the histogram of transformed data *)

ed1 = Select[Flatten[edMap], # > -9999 &];

hed = Histogram[ed1, ChartStyle -> Orange, ChartBaseStyle -> {Opacity[0.8]}]

(* The response function is computed as follow: *)

tv = Table[i, {i, -1, 30, 1}];

pv = Map[CDF[edist, #] &, tv];

tvpv = Thread[{tv, pv}];

ListPlot[tvpv, Joined -> True, PlotStyle -> {Opacity[0.8], Orange}, PlotMarkers -> {\[FilledCircle], 15}, BaseStyle -> {FontSize -> 16, FontFamily -> "Arial"}]

(*==========END OF CODE ============*)

Key words: ASC file, Thrips palmi, standarization, binarization

References

McDonald JR, Bale JS, Walters KFA. 1999. Temperature, development and establishment potential of Thrips palmi (Thysanoptera: Thripidae) in the United Kingdom. European Journal of Entomology. 96: 169-173.