How to access CANFAR and install and use Skytree

The current Skytree official release version is 11.6

1. Register for an account at the Canadian Astronomy Data Centre: http://www.cadc-ccda.hia-iha.nrc-cnrc.gc.ca/auth/register.html .

2. Request a CANFAR account via request to CanfarHelp at nrc-cnrc.gc.ca . Please include a few sentences describing what you wish to utilize the system for.

3. Await manual setting up of CANFAR account by CANFAR sysadmin using CADC username.

4. Request SkytreeServer<version>.tgz and skytree-client.lic: nick.ball at nrc-cnrc.gc.ca .

5. Copy files to CANFAR:

scp -p SkytreeServer<version>.tgz skytree-client.lic <username>@canfar.dao.nrc.ca:~

6. ssh to CANFAR:

ssh <username>@canfar.dao.nrc.ca

7. Run CANFAR setup, answering yes to creating a .netrc. This creates a public ssh key required for creating a virtual machine:

canfarsetup

8. Create virtual machine (VM) for Skytree (supply desired name for VM):

vmcreate <vmname>

9. Await email confirming VM creation.

10. Copy files to VM (VM IP may be obtained from VM creation email, or vmlist):

scp -p SkytreeServer<version>.tgz skytree-client.lic <vm IP address>:~

11. ssh to the VM as root:

ssh root@<vm IP address>

12. Untar the Skytree Server package to /opt/skytree, and move the license into it, and copy over all contents to /opt/skytree (recommended location for install):

cd /home/<username>

mkdir /opt/skytree

tar -xzf SkytreeServer<version>.tgz --strip 1 -C /opt/skytree

mv skytree-client.lic /opt/skytree

13. Create a group for Skytree Server users:

groupadd skytree-user

14. Setup the appropriate privileges for the installed files:

chown root /opt/skytree

chgrp skytree-user /opt/skytree

chmod 750 /opt/skytree

cd /opt/skytree

chown -R root *

chgrp -R skytree-user *

chmod -R 740 *

chmod 750 skytree-server

chmod 750 dataprep

chmod 750 dataprep/*.sh

chmod 750 datasets

chmod 750 docs

15. Create symlinks to the installed executable files for direct access from shell for all users:

ln -s /opt/skytree/dataprep/* /usr/local/bin

ln -s /opt/skytree/skytree-server /usr/local/bin

16. Users need to be added to the skytree-user group:

usermod -g skytree-user <username>

17. Exit the VM:

exit

18. ssh as oneself to the VM:

vmssh <vmname>

19. Setup user environment variables.

For BASH users, add the following lines to /home/<username>/.bashrc:

export SKYTREE_LICENSE_PATH=/opt/skytree/skytree-client.lic

export CLASSPATH=$CLASSPATH:/opt/skytree/dataprep/skytree.jar

For TCSH users, add the following lines to /home/<username>/.cshrc:

setenv SKYTREE_LICENSE_PATH /opt/skytree/skytree-client.lic

setenv CLASSPATH $CLASSPATH:/opt/skytree/dataprep/skytree.jar

20. From any directory, test the installation by running the following commands:

skytree-server

generate-header.sh

Both the above commands should give options available to the user.

To see what algorithms are available, e.g., nn, type skytree-server --help. For help on each algorithm, type skytree-server <algorithm_name> --help.

VOSpace Storage System

To access CANFAR's data storage, VOSpace, either go to the CANFAR dashboard, for interactive access, or see the VOSpace page on the wiki.

The following steps (as on the wiki) work for setting up the Python client on the VM, which allows VOSpace to be viewed as a mounted filesystem:

sudo yum install fuse

sudo yum install fuse-devel

sudo /usr/sbin/usermod -a -G fuse `whoami`

Log out, log in again

sudo yum install python26

sudo yum install python26-distribute

sudo easy_install-2.6 vos

getCert

mountvofs

Now commands like

vls vos:<username>

work, and can be used in scripts (e.g., vcp).

Batch Mode

Follow instructions for processing as with any other software on CANFAR: https://wiki.cadc-ccda.hia-iha.nrc-cnrc.gc.ca/canfar/index.php/Introduction .

Skytree can be called from scripts in the same way as it is run interactively.

Assuming you have VOSpace set up as described above, to run an example batch job, do the following:

1. Download these 4 files: run_skytree_example.sh, skytree_example.condor, skytree_example.sh, sdss100kx4.txt .

2. Place run_skytree_example.sh and skytree_example.condor on the CANFAR head node, and skytree_example.sh on your VM, each one in your home directory.

3. Place sdss100kx4.txt in your home directory on VOSpace via "Manage Storage" on the CANFAR dashboard (left-hand navigation bar -> add files).

4. Edit the two .shs and the .condor to change <username>, <vmname>, and <user email> to your own as appropriate.

5. Make sure each .sh is executable (chmod 755, or similar).

6. Shut down your VM (vmstop <vmname>, from the head node, and wait for the email).

7. Make sure your VOSpace and VM are public:

Go to your VOSpace home directory on the dashboard as in step 3

Click the two dots at the top

Click the icon next to your username

Edit permissions to public

Do the same for the .img.gz of your VM in the top level of your VOSpace

8. From the head node, submit the job:

condor_submit skytree_example.condor

It should output the files distances.out and indices.out to your VOSpace, and a logfile, stdout and stderr to the files indicated in the .condor. You can monitor the job with condor_q <username>.