Globus Flows

Globus provides some “Flows” in their library (check Globus Flows page), and many others are “published” by users or institutions for use. Check Globus Github: https://github.com/globus for more information.

Globus Flows using Globus Portal

• Open the Globus Portal (https://www.globus.org/) and select “Flows” on the left pane.

• Click on the library tab, search for Move (Copy and Delete) Files, and click “Start” button on the right pane. 

  • If you want to schedule a job, click on “View Runs for the Flow”

• Fill out the required fields and then click “Start Run. Also, check the “Advanced” tab where you can paste your input file in JSON format. 

• The components of flows (Check the “Definition” tab):

  • definition.json - the flow definition

  • schema.json - the flow input schema

  • trigger_*.py - a Python script that will trigger the flow

Globus Flows using Globus CLI

• This flow will transfer the file from source (GCP) to destination (GCS) and delete the source directory

• ssh to the dtn node for Markov “ssh markov-transfer” if you are transferring files directly (e.g. using scp, rsync etc.)

• Find the definition file “move-delete-only-def.json” at ~/usr/local/doc/GLOBUS - 

• Load the appropriate python module with your virtual environment (Check Python Virtual Environment for details)

• Activate/source the virtual environment and install globus cli package

  • source ~/<path-to-virtual-environment>/bin/activate

  • pip3 install globus-cli 

• Create a flow with a name “flow-test-recent “ using the definition file. The output prints the flow ID. 

  •  globus flows create flow-test-recent ./move-delete-only-def.json  # check Globus Flows CLI commands

• You will be prompted for authorization for the first time (one time only)

• Start the flow with the meaningful label. It will print the “Run ID”. 

  • globus-flow]$ globus flows start <flow ID> --label flow-cli-test

• For the first time, you may need to run “globus flows run resume” for granting access to the directories

  • globus flows run resume <Run ID>

• Check the Status of the flow. 

  • globus flows run show <Run ID>

• If you made some changes in the definition file, reflect the update with:

  • globus flows update --definition move-delete-only-def.json <flow ID>

• Check the list of flows. It will show the list of flows along with their Flow IDs.

  • globus flows list

• Delete the flow

  • globus flows delete <Flow ID>