LogMaker 4 GoogleDocs

What is the LogMaker

LogMaker is an interface to quickly

generate experiment logs in google drive (e.g. for high power runs),
add scan tables or other repetitive templates like daily logs with MC database info, date, time etc.
finish your document with an epilog, e.g., after scans are finished.

Video

A video with closed captions (turn ‘em on) can be found on youtube. In this example, the plugin has been implemented in the HTT CTRL Center.

Notes

You need to use your LBL account for this to work. For the first time you use it, and once every blue moon, google will prompt you to login and authorize the program.
Scan number ‘0’ is the ‘auto’ mode, looking for the newest scan.
The LogMaker requires the latest version of MasterControl based on LV2020 (with corresponding runtime), since it uses the newly formatted ECS dump file.

Adapting LogMaker for an experiment

Easy

Follow the steps below. This will allow you to do the things in the video with your own templates and your own scanlogs.

Advanced

If you want to dive deeper, or for further information, check the section ‘Backend and code modifications’ below.

How to get LogMaker in <10 minutes

Files and Folders on Google Docs

Template Files

The Logmaker in its current version uses three google docs templates to assemble new experiment log documents step-by-step. It then replaces placeholders with time/date, data obtained from MC database and optional screenshots. So the first step is to make a new gDrive folder for templates and three templates as follows:

The document template with the date, TOC, some items we have to do before a HP run. Example:

Note: this template is fully customizable. The {{MM-DD-YY}} placeholder will be replaced by the LogMaker with the current values.

2. The scan table template with placeholders for data you want to retrieve from MC database. Example:

Note:

- The few specific placeholders here in this example are hardcoded (that is, unless you go to the advanced section later in this document and mess around with it). E.g., the scan number will always be {{Scan no}}.
- Placeholders are not case sensitive.
- You can define additional placeholders, e.g., for a specific device parameter (e.g., stage positions). Placeholders need to be put in double parentheses {{example}}. They will be linked to the specific device parameter in the placeholders.ini file as explained in the Configuration section below.
- The current scripts look whether the scan you want to append to your log is already present (to avoid double postings). This search uses the “MM-DD-YY.*Scan Scan{{Scan No}}:”, which you might want to keep (or remove) for this reason. If you want to change the search phrase, refer to the Backend and modifications section below.

3. The Epilog. E.g., for steps that need to be done after a long night of scans. Example:

Google Drive Folders

You will need three google folders:

The folder that contains your templates
The folder that will contain your resulting experiment log
The folder that contains the optional screenshots

Note: these folder should be accessible/shared to everybody who is supposed to use your LogMaker.

Configuration

Make a copy/clone of the GEECS-Plugins repository on github https://github.com/ag6520/GEECS-Plugins. For GEECS developers, this folder could be in C:/GEECS/Developers Version/GEECS-Plugins. Typically, the GEECS-Plugins directory should be in the Experiment folder next to the Active Version of GEECS.
Copy the google credential files from vol1/software/control-all-loasis/HTW2020/Active Version/GEECS-Plugins/LogMaker 4 GoogleDocs/source/credentials.json to your LogMaker source folder.
The LogMaker build folder should contain four *.ini files. Make copies of those files in the user data folder (or a subfolder therein). Now, you will modify two of these copied files.

parameters.ini

One of the files will have ‘parameters’ in its name. Open this file with a text editor.

From the previous steps you will have three template files and three google drive folders.

In the parameters file, you will enter the document and folder google IDs that you can retrieve from the URLs like so:

logtemplateid -- ID of the base document template

scantempalteid -- ID of the scan table template

endtemplateid -- ID of the Epilog template

templatefolderid -- ID of the folder containing your templates

logfolderid -- ID of the folder containing your new LogMaker experiment logs

screenshotfolderid -- ID of the folder containing optional screenshots. If you do not plan to use screenshots you may put your logfolderid or templatefolderid here

logid -- handled by the Logmaker, no need to change

databasepath -- basepath to the MC data

pathtoscreenshot -- path to your optional screenshots

logname -- name of your log documents (will be preceded by the current date in the title of the generated logs)

Screenshot: The values ml, mt, mr, mb and imgscale refer to margins for left, top, right and bottom to cut your image, and imgscale to scale your figure respectively.

Scannumber: This value should be 0 by default (in this mode, the program looks for the latest scan automatically). It can be modified (e.g., directly in the file or through LabView) to make tables for specific scans (in which case scannumber !=0).

placeholders.ini

Pretty much explains itself. For first tests, you do not need to modify this file.

The left side identifies placeholders that can be used in the google docs. The few hardcoded placeholders (cf. scan table example above) do not appear in this file.

Double parenthesis {{}} are used in Gdocs templates to identify placeholders (check templates above), but are dropped in this file for convenience. Placeholders are not case sensitive.

The right side of the equation links the placeholder with its master control device and parameter. The && separates device and parameter. This ini does not use MC aliases.

In the example, I could write {{jet x}} anywhere in a template to replace it with the current value of Device HTT-ESP01 Parameter Position.Axis 1 in each new copy of the template added to my experiment log.

Other ini files (that you don’t need to modify)

Just for completeness, what’s going on in the other .INI files? You will not have to modify those

Config.ini contains the google ID of the google apps script that is called by the api.
Currentvalues.ini contains the latest values for all custom placeholders and hardcoded standard placeholders

Running LogMaker executables

You’re done. Now you just have to run it!

Python

If not done already, install python on your computer. The scripts have currently been tested up to version 3.9.

Install the following required packages into python (enter the code in your cmd line)

pip install pillow

pip install google-api-python-client

pip install google-auth-httplib2

pip install google-auth-oauthlib

[and everything I might have forgotten]

Open a windows cmd. Navigate to your copy of the LogMaker build. You’ll find four .exe files.

Run the scripts like so:

python appendScan.py parameters.ini placeholders.ini currentvalues.ini

You have to login to google for the first time executing your script. Use your LBL account.

You'll have to specify the full paths to your .ini files that are now located in the user data folder (or a subfolder therein).

Notes:

the config.ini needs to be in the same folder as the scripts. All other inis can be located anywhere, you can rename them to your liking and you can specify their full paths. It could be convenient to have them in the user data folder.

appendScan will generate a new ScanLog for a new day, if there is no Log present within the Experiment Log folder on Google drive yet. And then attach a new scantable for the latest scan.
appendScanOldLog will instead append a ScanTable to the last Scanlog on record (i.e., google ID saved in logID in parameters.ini file) without generating a new log. This allows scandays extending beyond midnight.
createGdoc generates a new log for the day, if this does not exist yet.
appendEpilog appends the Epilog template to the Experiment Log

Pro Tip: Calling the same LogMaker scripts with different sets of ini files, allows you to have a variety of Experiment Logs (e.g., different templates) for the same experiment.

Pro Tip: Running these scripts from the command line enables you to write your own wrappers for these functions. You could for instance make your own python script that stays open to detect new scans, and to append scantables automatically whenever a new scan is detected. You can also use them from within LV (check example from HTT CTRL in the intro video)

Backend and code modifications

For a simple description of required *.ini files and their meaning, please refer to the section ‘Templates and configuration’.

If you want to change and recompile the scripts or write your own functions, this is probably the right section to look at. All python code has been written and tested up to Python 3.9

Note on version control

LogMaker is maintained as part of the GEECS system at BELLA Center, LBNL. Specifically, it is part of the GEECS-Plugins github repositories. In order to work in the source code, first get the latest versions from github. https://github.com/ag6520/GEECS-Plugins. The local development folder should be located in C:GEECS/GEECS-Plugins/LogMaker4GoogleDocs.

Note that credentials are not supposed to be synced to github and are excluded in the gitignore file. If it happens anyway, please notify Tobi.

Parts of LogMaker

Docgen module
Google apps script in HTT logmaker project running in google cloud
Python scripts. Use docgen module and google apps script functions

The local side: docgen python module

This python module contains functions using google's apps script api to perform all relevant functions in your google doc. It also contains functions to interface with GEECS master control data. Most functions are ‘somewhat commented’ and documentation is an ongoing process. A current documentation can be found in docgen.html within the LogMaker4GoogleDocs source directory.

The server side: Google API and apps script

Some docgen functions make calls to the google api (apps scripts and drive). The server-side code is an apps script that implements functionalities and interacts with your document online. The apps script runs online in the google apps script project. It is deployed as an LBL internal api, so use an LBL account whenever asked for logging in.

What happens: the docgen python functions take whatever you give them as input (e.g. from Master Control scans) and form json api requests that are sent to the google apps script. This script performs the requested functions in your google doc and can also return info (e.g. strings, always within a json return file) to the python code.

A copy of the apps script code (Code.gs) can be found in the github repository, in the LogMaker4GoogleDocs source directory.

The LogMaker python scripts (example)

The python scripts (appendScan, appendScanOldLog, createGdoc, appendEpilog). They import all relevant modules, load your ini files, define environment variables and call a defined set of docgen functions to get data from master control and interact with google. Let’s go through an example:

Note: the code has changed (improved) significantly in certain places since the documentation below was written. But you'll get the hang.

1. Importing modules, reading the ini-files (lines 24-26) and loading their info into variables.

2. Still loading ini data. But also:

Lines 73-80: Clear existing currentvalues.ini file and build the new currentvalues.ini that will contain the default and special placeholders with their latest scan data.

Line 63: specifying a logfilename for google docs

Line 94-98: build the folderpaths to this date’s scan and ECS live dump folders as produced by master control (lines 94-98).

3. Some logic to allow for posting of old scans if specified in parameters.ini, and to avoid double-posting of scans in the same log

4. Execute: series of api calls and some basic logic (see below)

Line 126:

Establish service with google apps script api. The service variable will be passed on to the other functions called in this script.

Line 127:

createExperimentLog. This apps scripts function will create a new google doc for your experiment log, if none is there for the current day. If a document with appropriate name is in the google folder you dedicated to the experiment logs, it will not overwrite this file nor generate a new one.

The function returns the google ID of either the newly created logfile, or of the logfile for this current day that was found in the google folder.

This ID is stored in the DOCUMENT_ID variable for further use. It is also stored in the parameters.ini as logid.

Line 128:

Appends the TEMPLATE_ID (in this case the scantable template) to the google doc with the DOCUMENT_ID. However, if the search term (built in step 3) is already present in the document, it will not append a template (avoiding double posting of the same scan).

Lines 129-134:

Generate the currentvalues.ini file with all the default and specifically defined placeholders and their values retrieved from master control folders. The if/else case is for default (0, latest scan) or (!=0, an older scan). The if/else case here is thus directly related to lines 107-119 above.

Line 135:

Find and replace all placeholder names in the google document with ID DOCUMENT_ID and replace them with their current values.

Lines 136-139:

Cut and scale an image according to the INI specifications for screenshot. Upload the image to the screenshots folder in google drive. Find the placeholder {{screenshot}} in the google docs and replace it with the image. Currently the placeholder for the image must be in a Table.

Lines 140-144:

Handle cases where no screenshot is present or the screenshot is too small to cut your margin. If no screenshot is being uploaded, the placeholder {{screenshot}} in the document will be removed from the document, such that there won’t be multiple such placeholders in the doc when the next screenshot comes to replace them all at once.

The python scripts can be run and tested from within your GEECS/Plugins/source/Plugins folder as follows:

python appendScan.py parameters.ini placeholders.ini currentvalues.ini

Make sure to use the full paths to your ini files.

If you add substantial new functionalities, please add descriptions to this website.

Page updated

Report abuse