Batch Processing

Batch Processing

JOSE LOPEZ-COLLADO

July 2015

Often times it is necessary to process many files applying the same function(s). Here, I present an example of processing a batch of files representing maps with the goal of computing the proportion of a target area in relation to the whole area.

The input files correspond to the potential distribution of three rodent species (Oryzomis couesi, Sigmodon arizonae and S. hispidus toltecus), estimated with MaxEnt.

We can save the output as a list of files, depending on their types; in the example, image files (png) are saved.

Additional operations are presented to compute species diversity and co-occurrence, that is, the union of the maps (sum) and the intersection (multiplication).

The steps are as follow:

Define a function to binarize the matrices (see Basic Raster Operations)

Define a module (function) to compute the number of positive elements (1s), the raster size and the proportion

Get the list of files of a given type (asc in the example)

Load the files

Binarize the files

Compute total of target area and proportion

Generate list of figures and make a composite chart

Generate a matrix of file names and computed statistics, sort in descending order

Create a Bar Chart

Process the file list to change the names and save figures as png

Sum the raster to obtain the species diversity (0, 1, 2, 3 species)

Multiply the raster to obtain the co-occurrence of the three species in the same location

Process the rasters to substitute (replace) the augmented null values resulting from the operations.

Notice:

The example files are in the File Box page

Here is the code with comments, which contain the output as well:

(*=================== START CODE ==================*)

SetDirectory[NotebookDirectory[]];

(*

Define function to binarize maps

*)

binmap[xmap_, minx_, maxx_, nullv_] :=

If[xmap == nullv, nullv, If[(xmap <= maxx) && (xmap >= minx), 1, 0]];

SetAttributes[binmap, Listable];

(*

Define a function to compute the target area as a fraction: The

function has two arguments, x is the matrix and nullv is the null

value, first selects the vector of valid names, Total computes the

sum of 1s, d is the size of the vector, N[tot/d] is the fraction;

The module returns: the total, d is the size of vector and the fraction of 1s values:

assuming input matrix contains null values, 0 and 1s

*)

calcA[x_, nullv_] := Module[{vec, tot, perc, d},

vec = Select[Flatten[x], # > nullv &];

tot = Total[vec]; d = Flatten[Dimensions[vec]];

Flatten[{tot, d, N[tot/d]}]

]

*===============================================*)

(* ==================Read files ======================*)

(* FileNames scan the working directory for files with given

extension *)

ascf = FileNames["*.asc"]

{"Oryzomys_couesi.asc", "Sigmodon_arizonae.asc",

"Sigmodon_hispidus_toltecus.asc"}

(* Import files and truncate first six header lines*)

spp = Map[Drop[Import[#, "Table"], 6] &, ascf];

(*============================================*)

(*================== Binarize files ==================*)

binspp = Map[binmap[#, 0.2, 1, -9999] &, spp];

(*========================================*)

(*============ Compute total 1s and fraction =========*)

c1s = Map[calcA[#, -9999] &, binspp];

c1s

{{235531, 624342, 0.377247}, {266063, 624342,

0.426149}, {51653, 624342, 0.0827319}}

figs =

Map[ArrayPlot[#, PlotRange -> {Full, Full, {0, Automatic}},

ColorRules -> {0 -> Gray, 1 -> Orange}, ImageSize -> 200] &,

binspp];

compositeFig = GraphicsColumn[figs]

(*Make a bar chart of fractions *)

names = StringSplit[ascf, "."][[All, 1]]

{"Oryzomys_couesi", "Sigmodon_arizonae", "Sigmodon_hispidus_toltecus"}

(*Make a list with names and statistics*)

vn = Thread[{names, c1s}]

{{"Oryzomys_couesi", {235531, 624342,0.377247}}, {"Sigmodon_arizonae", {266063, 624342,0.426149}},{"Sigmodon_hispidus_toltecus", {516Co53, 624342,0.0827319}}}

(*Flatten the list so to have a 3 x 4 matrix*)

vnf = Map[Flatten[#] &, vn]

{{"Oryzomys_couesi", 235531, 624342,0.377247}, {"Sigmodon_arizonae", 266063, 624342,0.426149}, {"Sigmodon_hispidus_toltecus", 51653, 624342, 0.0827319}}

(*Sort and reverse order to make a chart*)

vns = Reverse[SortBy[vnf, Last]]

{{"Sigmodon_arizonae", 266063, 624342,

0.426149}, {"Oryzomys_couesi", 235531, 624342,

0.377247}, {"Sigmodon_hispidus_toltecus", 51653, 624342, 0.0827319}}

(*The bart chart uses labels instead of using the names from the matrix of values because of the size of the strings*)

bc = BarChart[vns[[All, 4]], ChartLabels -> {"SARZN", "OCOUS", "SHTOLT"}, BaseStyle -> {FontSize -> 12, FontFamily -> "Verdana"},ChartStyle -> 12]

(*==================================================*)

(* Export individual figures *)

(* change name of files to png and export *)

(*==================================================*)

newfilenames =

Map[StringReplace[ToString[#], ".asc" -> ".png"] &, ascf]

{"Oryzomys_couesi.png", "Sigmodon_arizonae.png","Sigmodon_hispidus_toltecus.png"}

MapThread[Export[#1, #2] &, {newfilenames, figs}];

(*========= Compute species diversity occurrence and co-occurrence ====*)

(* this is the number of species occurring at each cell value:*)

divSpp = Sum[i, {i, binspp}];

(*This is where the three species overlap: *)

overlapSpp = Product[i, {i, binspp}];

(*Next lines identify the null values as negative*)

DeleteDuplicates[Flatten[divSpp]]

{-29997, 0, 1, 2, 3}

DeleteDuplicates[Flatten[overlapSpp]]

{-999700029999, 0, 1}

(* Replace for a new null value= -1, use {2} to apply at level 2 of the matrix *)

spD = Replace[divSpp, -29997 -> -1, {2}];

(*

Show the map with the three species

o is when no species occur, 1 for one species, 2 for two species, 3 for three species

*)

ArrayPlot[spD, PlotRange -> {Full, Full, {0, Automatic}},

ColorRules -> {0 -> Gray, 1 -> Blue, 2 -> Yellow, 3 -> Red},

ImageSize -> 450]

(* replace null values for the co-occurrence species and show the map *)

ovSp = Replace[overlapSpp, -999700029999 -> -1, {2}];

ArrayPlot[ovSp, PlotRange -> {Full, Full, {0, Automatic}},

ColorRules -> {0 -> Gray, 1 -> Black}, ImageSize -> 450]

(*================END OF CODE ===============*)

key words: Mathematica, file batch processing, binarize maps, bar chart, target area, species co-occurrence