+ High Performance Super Computing

Java and MPI wrapper compilers (Open MPI and MPICH) are also available.

For more information, see "What compilers are available on the IU research systems?" kb.iu.edu/d/abby

Submitting and Running Jobs

Job Management is provided by the TORQUE resource in combination with the Moab job scheduler. Batch scripts will need to be configured for the specific system upon which the script will run.

A TORQUE batch job script for an MPI job no Karst might look like this:

#!/bin/bash

#PBS -l nodes=2:ppn=16,walltime=30:00

#PBS -M jthutt@tatooine.net

#PBS -m abe

#PBS -N JobName

#PBS -j oe

mpirun -np 32 -machinefile $PBS_NODEFILE a.out

For more about job scripts, see "What is Torque, and how do I use it to submit and manage jobs on high-performance computing systems?" kb.iu.edu/d/avmy

EXAMPLE JOB CONTROL COMMANDS

Submit script:

qsub my_job_script.pbs

Query scheduler:

showq -i

Query job status:

qstat -u username

Delete a submitted job:

qdel job_id

For more about scheduling, see "What is Moab?" kb.iu.edu/d/avmu

Note: Big Red II is made up of two parts: the external service nodes and the actual compute nodes where the jobs run. The figure to the right shows that all jobs on Big Red II will first land on an "aprun" service node. Jobs require unique commands to launch jobs on Big Red II compute nodes. In general, use the "aprun" command to launch your jobs, whether they are serial or parallel.

For more information, see "Execution environments on Big Red II at IU: Extreme Scalability Mode (ESM) and Cluster Compatibility Mode (CCM)" kb.iu.edu/d/bdol and "How do I run batch jobs on Big Red II at IU?" kb.iu.edu/d/bdkt

System Specifications

For more information about queues, see "Queue information for IU research and computing systems" kb.iu.edu/d/bdkd

Interactive Jobs

In addition to fire-and-forget jobs, you can interactively run jobs on the compute nodes. 

To run interactive jobs, use "qsub -I" along with other command line arguments to describe your job needs,

On Karst, use the interactive queue. For example:

qsub -I -q interactive -l nodes=1:ppn=1,vmem=10gb,walltime=4:00:00

To use X forwarding in an interactive job, use the -X flag. For example, on Karst:

qsub -I -X -l walltime=00:45:00 -l nodes=1:ppn=4 -l vmem=10gb

For more information about X forwarding, see "On my personal computer, how do I use X forwarding to securely run graphical application installed on Big Red II, Karst, or Mason at IU? kb.iu.edu/d/bdnt

Home Directories

When you create an account on one of IU's research computing resources, you are allotted 100 GB of home directory disk storage. The 100 GB quota is shared across all accounts on IU machines. 

An example path to you home directory is as follows: /N/u/<username>/BigRed2

For more about home directories, see "At IU, how much disk space is available to me on the research computing systems?" kb.iu.edu/d/avkm

Data Storage Systems

Data Capacitor II File System

Data Capacitor II (DC2) is a large-capacity, high-throughput, high-bandwidth Lustre file system serving all IU campuses. It is mounted on Big Red II, Karst, and Mason.

For more about DC2, see "At IU, where is my scratch or project directory on the Data Capacitor II or DC-WAN file system, and how do I view directory permissions and determine how much space I"m using?" kb.iu.edu/d/axod

You can access your storage space on DC2 at /N/dc2/scratch/<username>. This space is not intended for permanent storage of data and is not backed up.

For more about high-speed file systems, see "The Data Capacitor II and DC-WAN high-speed file systems at Indiana University" kb.iu.edu/d/avvh

USEFUL LUSTRE COMMANDS

Check storage space usage (get a total sum of data stored in your scratch directory)

du -hc /N/dc2/scratch/<username>

Find oldest files (number list files in reverse order by date modified)

$ cd /N/dc2/scratch/<username>

$ find . -type f -exec ls -hltr "{} +;y

For more, see "About Lustre file systems" kb.iu.edu/d/ayfh

The Scholarly Data Archive

The Indiana University Scholarly Data Archive (SDA) provides extensive capacity (42 PB) for storing and accessing research data. Files that need to be kept for a long time but are not frequently updated are best suited for storage here.

Methods available for transferring data to and from the SDA include Kerberos-enables FTP, parallel FTP (pftp_client), Hierarchical Storage Interface (HSI), secure FTP (SFTP), secure copy (SCP), SMB/CIFS/Windows file sharing (SMB), and https (via a web browser).

For more information about the SDA, see "The Scholarly Data Archive (SDA) at Indiana University" kb.iu.edu/d/aiyi

IU Cyberinfrastructure Gateway

https://cybergateway.uits.iu.edu/iugateway

The IU Cyberinfrastructure Gateway provides a centralized web interface where you can:

• • Monitor you usage of IU research computing and storage resources: jobs/month, core hours/month, queue time/month, storage on RFS and SDA, etc.

  • Move data from desktop of laptop workstations to the IU SDA and DC2

  • Find information about available scientific and bioinformatics applications