Cryo-EM software installation on GoToCloud

1.1. Create ParallelCluster

Log in to your AWS account with administrator and create ParallelCluster.

For more information, see: https://sites.google.com/sbrc.jp/gotocloud-doc/procedure

 [2024/10/02 Developer's Note] When updating EFS in the KEK system, Cloud9 should be created in the usr-vpc as per the hands-on instructions. Only kek901 is peered with the master-vpc, allowing updates to the master-EFS. Ideally, it should be named something like master-operate-vpc, rather than usr-vpc.

1.2. Assign EFS write permission

Search for EC2 service in search window and navigate to EC2 service page. 

Select "Instances" form menues on the left.

Select the head node created in procedure 1.1. and select "Actions", "Security" and "Modify IAM role"  in that order from the top right box.

Choose "EFSWriteAccessRole"and click "Update IAM role".

5.1. Install crYOLO

The following instructions assume that anaconda are available. 

Connect to head node via ssh.

$ cd /efs/em/

$ export PYENV_ROOT=/efs/em/pyenv　

$ export PATH=$PYENV_ROOT/bin:$PATH

$ pyenv versions

  system

* anaconda3-2023.03 (set by /efs/em/pyenv/version)

$ pyenv global anaconda3-2023.03

$ pyenv version

anaconda3-2023.03 (set by /efs/em/pyenv/version)

$ export PATH=$PYENV_ROOT/versions/anaconda3-2023.03/bin/:$PATH

create conda environment and install cryolo

$ conda create -n cryolo-1.9.6 -c conda-forge -c anaconda pyqt=5 python=3 numpy=1.18.5 libtiff wxPython=4.1.1 adwaita-icon-theme 'setuptools<66'

$ source activate cryolo-1.9.6

$ pip install nvidia-pyindex

$ pip install 'cryolo[c11]==1.9.6'

$ conda deactivate

・When conda create takes too long time
Change the package solver of conda to libmamba.
cf. https://www.anaconda.com/blog/a-faster-conda-for-a-growing-community

$ conda install -n base conda-libmamba-solver

$ conda config --set solver libmamba

<Just in case> How to remove conda environment

$ export PYENV_ROOT=/efs/em/pyenv　

$ export PATH=$PYENV_ROOT/bin:$PATH

$ export PATH=$PYENV_ROOT/versions/anaconda3-2023.03/bin/:$PATH

$ conda env remove --name cryolo-1.9.6 

5.2. module setting for crYOLO

Create module file.

$ cd /efs/em/modulefiles/crYOLO

$ cat > 1.9.6

#%Module -*- tcl -*-

set root /efs/em/pyenv/versions/anaconda3-2023.03/envs/cryolo-1.9.6

setenv CRYOLOPATH $root

prepend-path PATH $root/bin

setenv CRYOLO_SUBMIT_CMD sbatch

setenv CRYOLO_SUBMIT_SCRIPT /efs/em/aws_slurm_cryolo.sh
[CONTROL+C]

Setup default version.

$ cd /efs/em/modulefiles/crYOLO/

$ cat > .version

#%Module 1.0

set ModulesVersion 1.9.6
[CONTROL+C]

5.3. Download the general models

$ cd /efs/em/cryolo

$ mkdir models

Download all general models for crYOLO to head node from here. It is recommended to connect to head node via niceDCV and download from firefox via http.

Move all models for crYOLO to "/efs/em/cryolo/models" derectory

$ mv /path/to/model /efs/em/cryolo/models/ 

Create symbolic links for each models

$ ln -s /efs/em/cryolo/models/gmodel_phosnet_202005_N63_c17.h5 /efs/em/cryolo/gmodel_phosnet_lpf_link.h5

$ ln -s /efs/em/cryolo/models/gmodel_phosnet_202005_nn_N63_c17.h5 /efs/em/cryolo/gmodel_phosnet_denoise_link.h5

$ ln -s /efs/em/cryolo/models/gmodel_phosnet_negstain_20190226.h5 /efs/em/cryolo/gmodel_phosnet_negstain_link.h5

5.4 Install python scripts to run crYOLO using relion external job

$ cd /efs/em/

$ git clone https://github.com/KEK-SBRC-CryoEM/gotocloud.git

$ cp -r gotocloud/cryolo ./

6.1. Install Topaz     

If you have not installed pyenv and anaconda, perform step2. before installing Topaz.

Connect to head node.

If continuing from 3.1., the following is not required.

$ cd /efs/em/

$ export PYENV_ROOT=/efs/em/pyenv　

$ export PATH=$PYENV_ROOT/bin:$PATH

$ pyenv versions

  system

* anaconda3-2023.03 (set by /efs/em/pyenv/version)

$ pyenv global anaconda3-2023.03

$ pyenv version

anaconda3-2023.03 (set by /efs/em/pyenv/version)

$ export PATH=$PYENV_ROOT/versions/anaconda3-2023.03/bin/:$PATH

create conda environment and install Topaz

$ conda create -n topaz-0.2.5 python=3.6

$ source activate topaz-0.2.5

$ conda install topaz -c tbepler -c pytorch

$ conda deactivate