Globus provides some “Flows” in their library (check Globus Flows page), and many others are “published” by users or institutions for use. Check Globus Github: https://github.com/globus for more information.
Open the Globus Portal (https://www.globus.org/) and select “Flows” on the left pane.
Click on the library tab, search for Move (Copy and Delete) Files, and click “Start” button on the right pane.
If you want to schedule a job, click on “View Runs for the Flow”
Fill out the required fields and then click “Start Run. Also, check the “Advanced” tab where you can paste your input file in JSON format.
The components of flows (Check the “Definition” tab):
definition.json - the flow definition
schema.json - the flow input schema
trigger_*.py - a Python script that will trigger the flow
This flow will transfer the file from source (GCP) to destination (GCS) and delete the source directory
ssh to the dtn node for Markov “ssh markov-transfer” if you are transferring files directly (e.g. using scp, rsync etc.)
Find the definition file “move-delete-only-def.json” at ~/usr/local/doc/GLOBUS -
Load the appropriate python module with your virtual environment (Check Python Virtual Environment for details)
Activate/source the virtual environment and install globus cli package
source ~/<path-to-virtual-environment>/bin/activate
pip3 install globus-cli
Create a flow with a name “flow-test-recent “ using the definition file. The output prints the flow ID.
globus flows create flow-test-recent ./move-delete-only-def.json # check Globus Flows CLI commands
You will be prompted for authorization for the first time (one time only)
Start the flow with the meaningful label. It will print the “Run ID”.
globus-flow]$ globus flows start <flow ID> --label flow-cli-test
For the first time, you may need to run “globus flows run resume” for granting access to the directories
globus flows run resume <Run ID>
Check the Status of the flow.
globus flows run show <Run ID>
If you made some changes in the definition file, reflect the update with:
globus flows update --definition move-delete-only-def.json <flow ID>
Check the list of flows. It will show the list of flows along with their Flow IDs.
globus flows list
Delete the flow
globus flows delete <Flow ID>