crYOLO
cryYOLO
CrYOLO [1] is a fast and accurate particle picking procedure. It's based on convolutional neural networks and utilizes the popular You Only Look Once (YOLO) object detection system.
Important Notes
If you have huge data files in TBs, you may want to submit your job directly from the working directory at your project storage without copying the files to the scratch space.
If you are using the scratch space, it is recommended to delete it once the output files are copied b back
Cryolo in HPC
Cryolo in HPC has been installed as a singularity container and the environment path CRYOLO is defined in singularity module.
Interactive Job Submission
Request a GPU node
srun -p gpu -C gpup100 --gres=gpu:1 --mem=16gb --pty bash
Load Singularity Module
module load singularity
Check the Conda env
singularity exec --nv $CRYOLO conda env list
output:
activate does not accept more than one argument:
['conda', 'env', 'list']
# conda environments:
#
base * /opt/miniconda3
cryolo /opt/miniconda3/envs/cryolo
Check the usage for cryolo_train.py:
singularity exec --nv $CRYOLO /opt/miniconda3/envs/cryolo/bin/cryolo_train.py
output:
Not a conda environment: /opt/miniconda3/envs/cryolo/bin/cryolo_train.py
usage: cryolo_train.py [-h] -c CONF [-w WARMUP] [-p PATCH] [-e EARLY]
[-g GPU [GPU ...]] [--warm_restarts]
Batch Job submission
Download the reference data as explained in Cryolo Reference Example [2]
Use this job file to run your job:
#!/bin/bash
#SBATCH --nodes=1
#SBATCH --cpus-per-task=1
#SBATCH --time=01:00:00
#SBATCH -p gpu -C gpup100 --gres=gpu:1 --mem=32gb
cp -r * $PFSDIR
cd $PFSDIR
module load singularity
#warm up the network
singularity exec -B /scratch --nv $CRYOLO /opt/miniconda3/envs/cryolo/bin/cryolo_train.py -c config_phosnet.json -w 5 -g 1
# Actual Training
singularity exec -B /scratch --nv $CRYOLO /opt/miniconda3/envs/cryolo/bin/cryolo_train.py -c config_phosnet.json -w 0 -e 10 -g 1
# Prediction
singularity exec -B /scratch --nv $CRYOLO /opt/miniconda3/envs/cryolo/bin/cryolo_predict.py -c config_phosnet.json -w model.h5 -i unseen_examples/ -o my_results
cp -ru * $SLURM_SUBMIT_DIR
rm -rf $PFSDIR/*
Submit the job
sbatch cryolo.job
You will find the log file at slurm-<jobid>
output:
...
Using TensorFlow backend.
28/28 [==============================] - 128s 5s/step - loss: 0.0173 - val_loss: 0.0447
Using TensorFlow backend.
...
134 particles are found in tmp_filtered/unseen_examples/TcdA1-0197_frames_sum.mrc ( 100 % )
1611 particles in total are found ( 12 seconds)
Deleted 81 particles as they were not fully immersed in the micrograph
Check the GPU utilization:
ssh <gpu-node> nvidia-smi -l 2 # for every 2 seconds
output:
Mon Aug 5 10:18:23 2019
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 418.56 Driver Version: 418.56 CUDA Version: 10.1 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 GeForce RTX 208... Off | 00000000:02:00.0 Off | N/A |
| 27% 28C P8 20W / 250W | 0MiB / 10989MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 1 GeForce RTX 208... Off | 00000000:03:00.0 Off | N/A |
| 29% 44C P2 251W / 250W | 10693MiB / 10989MiB | 75% Default |
+-------------------------------+----------------------+----------------------+
| 2 GeForce RTX 208... Off | 00000000:81:00.0 Off | N/A |
| 27% 28C P8 21W / 250W | 0MiB / 10989MiB | 0% Default |
References:
[1] Cryolo Home