Language for Grant Proposals

Introduction

The following facilities and personnel descriptions are for use in facilities or research computing IT support sections in grant applications.

Facilities, Equipment, and Other Resources

Portland State University has a high degree of centralization of IT resources and support for academic and research computing. The Office of Information Technology’s (OIT) Linux Applications Platform team in the Technology Infrastructure (TI) division provides central research computing support for Linux research computing systems and provides free research computing resources, expertise, and training to students, faculty, and staff. The research computing staff work directly with researchers to identify their research computing needs in order to facilitate their research goals. TI's Windows, Virtualization, Storage, and Backups team provides Windows-based research computing systems, enterprise storage for network file shares, and backups. TI’s Data Center Operations and Networking teams provide data center and networking infrastructure support for central research systems.

OIT Research Cyberinfrastructure

The Office of Information Technology provides a highly functional, widely used campus cyberinfrastructure that effectively serves many PSU campus research projects. This is a thoughtfully integrated environment with network-accessible file systems for home directories, lab network shares, scratch volumes, web directories, and SQL databases to facilitate research work, data storage, and data sharing.

Research Colocation Facility

PSU’s Office of Information Technology (OIT) manages a data center in the Fourth Avenue Building, which is currently used to house PSU’s Coeus cluster. This facility has sufficient available server racks needed to meet the space requirements of the proposed hardware. Racks in the data center are arranged in hot/cold aisles for warm-air extraction. A computer room air handler provides cooling for this facility, including a series of multistack dedicated heat recovery chillers. Two diverse utility power feeds power the data center, which is also protected by two 300 kVA Mitsubishi uninterruptable power supply (UPS) units and two static transfer switches for power redundancy. In case of catastrophic power failure, the data center is also backed by a turbine generator. The total power available to the facility is 240 kVA. A very early smoke detection apparatus (VESDA) with a dry water pipe system used in this facility. Hand-held Halotron fire extinguishers are also located on-site.

Computational Resources

PSU's Office of Information Technology (OIT) supports a number of computational systems that host a large range of scientific applications for computational mathematics, chemistry, biology, genetic analysis, hydrology, fluid dynamics, and GIS in addition to a suite of general purpose scientific and statistical software, compilers, and interpreters. Support for applications and end users is provided by OIT teams.

Coeus HPC Cluster

Coeus is a general purpose HPC cluster designed to address a broad range of computational requirements. Its estimated peak performance is over 110 TFLOPs, and it offers Intel OmniPath low-latency networking, 200 TB NFS scratch storage, and 800 TB Panasas parallel scratch storage.

• Two login nodes and two management nodes head nodes:

  • 2x Intel Xeon E2630 v4, 10 cores @ 2.2 GHz

  • 64 GB 2400 MHz RAM

• 128 compute nodes:

  •  2x Intel Xeon E2630 v4, 10 cores @ 2.2 GHz

  • 128 GB 2133 MHz RAM

  •  200 GB SSD

• 12 Intel Phi processor nodes:

  • Intel Xeon Phi 7210, 64 cores @ 2.2 GHz, 4 hyper-threads/core

  •  96 GB 2400 MHz RAM

  •  200 GB SSD drive

•  2 large-memory GPU compute nodes:

  • 2x Intel Xeon E2650 v4, 12 cores @ 2.2 GHz

  • 1x Nvidia V100 GPUs

  • 768 GB 1866 MHz RAM

  • 2 TB local storage

• 10 GPU nodes:

  • AMD EPYC 7520P, 32 cores @ 2.5 GHz

  • 4x Nvidia A40 or RTX A5000 GPUs

  • 2 TB NVMe local storage

• Data Transfer Node to support high-bandwidth data transfers:

  • Dual Intel Xeon E2650 v4, 12 cores @ 2.2 GHz

  • 256GB 2400 MHz RAM

  • 30 TB local disk storage in a RAID 6 array

• 2x storage servers for /home and /scratch storage volume in warm spare

  • 200 TB NFS scratch storage

  • Dual Intel Xeon E2650 v4, 12 cores @ 2.2 GHz

  • 768 GB of 1866 MHz Registered ECC DDR4 memory

  • 32 x 8 TB SATA drives in a RAID 6 configuration

  • 2 TB NVMe local storage

• Intel OmniPath high-performance network fabric

• 1 Gb ethernet cluster management and IPMI networks

• 100 Gb high-performance network fabric (GPU nodes)

High-Performance Storage

A scratch volume is provisioned on the Panasas ActiveStor Ultra parallel file system storage array consisting of three ASD-100 director nodes and 20 ASU-100 storage nodes. The total raw capacity of the array is 1,688 TB; half of that is provisioned as a scratch volume on the Coeus cluster. Panasas nodes have a 25 Gb connection into the top-of-rack (TOR) switch that has a 100 Gb uplink into the PSU network. The proprietary Panasas DirectFLOW client is deployed on all of the HPC nodes to provide parallel high-performance storage, and the Panasas supports NFS and SMB protocols that can be used by non-HPC clients to access their data. Data on Panasas is not backed up, but Panasas snapshots are used to provide basic data recovery functionality. Currently, we preserve four 6-hour snapshots, four daily snapshots, and four weekly snapshots.

PSU’s NetApp storage array is used for HPC data that is supposed to be kept beyond the runtime of the job. Multiple volumes are provisioned for the purpose of home directories, research projects, software repositories, and configuration directories using NFS to Coeus nodes as well as to some other endpoints to provide ease of access for data movement. Primary volumes are hosted on a FAS8700 NetApp array located in our primary data center while all of that data is also replicated to a FAS8200 located in our hot site where it could be brought back online in case of a primary data center outage. All of the data is protected by 4 months of NetApp snapshots as well as backups provided by our enterprise IT Commvault instance. The NetApp array has multiple link aggregations to the 25 Gb network, providing robust and redundant network performance.

Networking

The Portland State campus core network includes dual, redundant 10 Gb paths between all science and research buildings and the data center. These buildings have 1 Gb networking to the desktop. Wireless 802.11 b/g/n networking is available in common areas and classrooms at speeds up to 1 Gb. PSU's science DMZ provides an alternate 10 Gb path via Internet2 bypassing the campus firewall allowing high-speed data transfer on and off of campus. Two data transfer nodes are available for high-speed data transfer on campus and off of campus via Internet2. OIT’s Linux Applications Platform team maintains two perfSONAR servers for research network performance monitoring.

OIT Personnel and Services

Centralized research computing systems are supported by Office of Information Technology (OIT) personnel. This includes seven Linux systems administrators from the Linux Applications Platform team; Windows server and enterprise virtualization, storage, and backups administrators from the Windows, Virtualization, Storage, and Backups team; a network engineer from the Networking team; and a data center specialist from the Data Center and Ops team. OIT infrastructure teams provide after-hours on-call support for any critical hardware, performance, or security issues that arise.