Last update: D. Zhu 01/06/2016
XPP has developed a set of MATLAB functions and scripts for LCLS data extraction and analysis based on the Standard HDF5 files. MATLAB functions, tutorials, examples are stored in directory /reg/neh/operator/xppopr/xppcode/matlab/.
The structure of subdirectories is described in the following
tools: general matlab tools, not directly hdf5 or XPP related.
hdf5tools: general hdf5tools, not directly related by the XPP file structure, these tools are used by those in XPPana to read HDF5 files
XPPana: analysis tools, which use the XPP specific detectors, common analysis tools
In the following, the functionality of most relevant scripts in each sub-folder are briefly described. For more detailed help see the documentation section of the corresponding functions. A large number of functions may be still “under construction”, sorry for that but we are constantly updating.
The main functions for reading data are stored in XPPana, but they make us of general reader stored in hdf5tools and in tools. We recommend to add the entire directory structure to the matlab search path, e.g. by
addpath(genpath(/reg/neh/operator/xppopr/xppcode_bak/matlab))
LCLS PCDS provide Matlab license the licenses on some of the local machines. To check which machine has license available, type the following command from any of the psana nodes(ssh -X psana).
> /reg/common/package/scripts/matlic
The current installation is 2016a with four popular requested toolboxes: curve ftting, optimization, image processing, and signal processing. It is located at:
> /reg/common/package/matlab/r2016a/bin/matlab
SSH to a machine once license is located, and start matlab there. Remember to always ssh -X for graphics.
For questions, or ideas for improvement, please contact dlzhu@slac.stanford.edu
The matlab tools developed at XPP consist mainly of functions reading hdf5 files into a meaningful matlab format in order to facilitate developing own matlab functions for special applications. The data readers are presently separated into one main function, reading all point detector data, and several small reader-funtions for array and area detector data.
For generally used procedures of data analysis, some higher level functions have been developed.
Point detector data are those data with at most one data value per FEL event (shot). These can be diode readings, accelerator parameters, but also motor positions which are updated at few second period. As relevant data from an entire run can usually be kept in memory they can be read with a single function rdXPPdata which returns a matlab structure containing the data.
The function takes essentially the hdf5 filename and a string list of detectors to read as input arguments (type help rdXPPdata in matlab for full documentation). Each data value has its own timestamp as identifier. As this is not generally necessary to be read, timestamp reading has to be selected separately from the data in most cases. Reading timestamps is necessary for shot to shot comparison between e.g. beamline diode data and accelerator data, where occasionally single events are lost. Here, matching the timestamps on the microsecond accuracy can ensure that the data values correspond to the same event.
While basically always the entire point detector data of one datafile can be read at once, generally only a subset of area detector data can be kept in memory at the same time. Therefore, the array detector reading functions take the filename, the scan step (calibcycle) and an array of events (shots) to be read as input argument. This enables to read only data from actually analyzed events, when filtering for e.g. beam intensity, timing or electron beam energy by previously read point detector data.
The main functions here are rdYAG1data, rdAcquirisData, rdCSPADdata, rdOpaldata, rdCS140M, rdPrincetondata, etc.
In this section we describe a few examples from previous measurements for different type of data that are frequently used in data analysis. The tutorial folder contains now two subfolders, the StandardHDF5 and the LittleData. This page will mainly discuss the StandardHDF5 format and related analysis. The LittleData based analysis is described here.
IPIMB, BLD, and other point detectors
The most frequently seen data type is what we call shot-to-shot point detector data. The examples are IPIMB data from PIPS diodes or beamline intensity monitors, BLD data such as ebeam energy, phase cavity timing, FEE gas detector measurements.
CSPAD
Putting together the 32 tiles of the detector can be intimidating at the beginning but we have developed a few functions to put things into perspective.
See reference:
CS140
This is the two tile CSPAD module that is frequently used when some spatial resolution is needed for scattering/diffraction measurement but only a small q range is of interest. One delay scan analysis example can be found at
/reg/g/xpp/xppcode/matlab/tutorial/StandardHDF5/CSPAD/L634_Example_r198.m
The example data are in the same folder.
Time Tool
One example data of Bi scan with time tool correction is located at
/reg/g/xpp/xppcode/matlab/tutorial/StandardHDF5/Bi/xppc0114-r0287.h5
The preprocessed time tool information can be accessed using
>> data = rdXPPdata(filename, {'tt'});
One example for the data sorting Bi_example.m can be found at the same location.
Opal
1024X1024 12-bit CCD camera that reads out at 120 Hz, typically. One example is located at:
/reg/g/xpp/xppcode/matlab/tutorial/StandardHDF5/OPAL/xppm0110-r0226.h5
Where two OPAL cameras with imaging optics + YAG:Ce crystal were used to record the FEL spectra at the same time during a preliminary test. An example script for extracting the two image sequences: OPAL_example.m is located in the same folder.
Quartz
2048X2048 8-bit CMOS camera from Adimec, runs 120Hz full frame but has higher noise than the Opal.
Orca Flash 4.0
2048x2048 16-bit from Hamamatsu, currently runs 30 Hz full frame and can run 120Hz with regions of interest. The rolling shutter requires some understanding and special treatment sometimes for the analysis.
Recently we started providing a condensed version of hdf5 file as a starting point for preliminary quick data analysis. The Matlab examples are described here.
Sanghoon made this little HDF5 browser for matlab which can be a convenient way to check what's in the .h5 file of interest. It is located at /reg/g/xcs/xcscode/matlab/GUI/HDF_Viewer.m.