Cluster Description:
The PHASIS cluster will be used for pre-processing and final reconstruction of soft x-ray diffraction data for ptychographic imaging at the new COSMIC beamline, 7.0.1, at the ALS.  The data are generated by the ALS fastCCD at up to 400 MBytes per second and typical datasets are 10s of GB in size.  The 64 CPU threads of the PHASIS cluster are used for pre-processing individual diffraction snapshots while complete reconstruction jobs are executed across 16 nVidia GTX Titan, 4 nVidia K80, and 4 nVIDIA TITAN-X GPUs.  This scale of this cluster enables realtime feedback to the COSMIC microscope user and makes the use of diffraction data for imaging transparent.

Hardware Configuration:
  • Master node (1)

Phasis master node has dual-socket octa-core Intel Xeon E5-2680 @ 2.70 GHz processors (16 cores in total), 256 GB of physical memory, and 260 TB (raw) of storage.

  • Compute nodes (6) 
    Cluster GPU cores total:  78,392

n0000-n0003 compute nodes each have dual-socket octa-core Intel Xeon E5-2680 @ 2.70 GHz processors (16 cores in total), 4 Nvidia GeForce GTX Titan GPUs (4x2688), 256 GB of physical memory.    

n0004 compute node has Dual Xeon E5-2630 2.3GHz CPU 8 cores, total of 16 cores, 256GB memory, and 4 nVidia K80 GPUs (4x4992).      

n0005 compute node has Dual Xeon E5-2699V4 2.2 GHz CPU 22 cores, total of 44 cores, 256GB memory, and 4 nVidia TITAN X GPUs (4x3854)                             

Compute nodes are connected with each other through a high performance Mellanox 56 Gb FDR Infiniband switch.

  • Storage

260 TB (raw) of storage is shared across the whole cluster and beamline 7.0.1 through the master node.


Login Procedure:
Phasis cluster uses One Time Password (OTP) for login authentication.
  • Login server: phasis.lbl.gov
Software Configurations and Data Transfer:
Globus Online Endpoint:  #phasis
        Authenticate using OTP at login web page

SSH-FUSE mounts allowed using OTP authentication  (currently disabled 080416 due to security issues)
       [Linux or MAC]
         e.g.
        (1) sshfs username@phasis.lbl.gov:/global/groups/cosmic/data  mount_point
        (2) to unmount:   fusermount -u
        (3) users need to remember to unmount their fuse mounts before the Phasis master node is rebooted.

Software Module Farm (SMF)
See here
for instructions on using the Global SMF.

Scheduler Configuration:
Phasis cluster uses SLURM to manage jobs on the cluster.

Partition
Nodes Node List
QoS QoS Limit
batch 6 n000[0-5]
normal no limits

Software Configuration:
Phasis cluster uses Warewulf as the system management software.  It also uses Environment Modules to manage the cluster wide software installation.

Cluster Status:
Please visit here for the live status of Phasis cluster.

Additional Information:
Please use Service Now or send email to hpcshelp@lbl.gov for any inquiries or service requests.