High Performance Computing for Science and Engineering

As manufacturing processes approach the physical limits of transistor density, efficient code must exploit parallelism to scale with available computing resources. Scientific software developers must therefore adopt a “think parallel” mindset to solve complex problems across academia, industry, and society. This course introduces parallel programming and its relationship to computer architectures, with an emphasis on high performance computing. Students will develop experience with programming models such as OpenMP, MPI, and CUDA, applying these techniques in homework and a term project.

Prerequisites

The course assumes the student is comfortable reading and writing code in C++ and Python. Familiarity with Linux command line tools, particularly ssh and git, is also expected. While advanced C++ knowledge is not required, CS 2050 will not teach the basics of programming, and the class is likely unsuitable for first-year students.

The course also assumes a working knowledge of mathematics used throughout science and engineering, including complex numbers, calculus, and linear algebra.

A determined student who lacks one or two of these prerequisites might still manage to succeed in the course, but only if they are willing to work hard and use external resources to fill gaps in their knowledge.

Course Objectives

• For students to develop a theoretical and practical understanding of parallel programming models, including those appropriate for shared memory computing (e.g., OpenMP), distributed computing (e.g. MPI), and accelerated computing (e.g. CUDA).

• For students to deepen their programming skills, with special emphasis on tools relevant for cutting edge scientific computing. The exercises and assignments will involve regular use of a compute cluster dedicated to the course. Students will make frequent use of git for managing source code, slurm for submitting jobs to the cluster, CMake for building software,  and further tools for profiling and optimizing code.

Course Format

The course elements are listed here, with more details below. Percentages in brackets indicate the contribution to the final grade.

1. Lectures: Main course content, supplemented by interactive demonstrations.

2. Quizzes [10%]: Short quizzes due weekly on Tuesdays.

3. Homework [40%]: Assignments due biweekly on Thursdays.

4. Project [25%]: Course project due at the end of the semester.

5. Final Exam [25%]: Comprehensive exam covering material from across the course.

Lectures

Lectures will be recorded and made available from Canvas.

Quizzes [10%]

Weekly quizzes (on Tuesdays) will be administered through Canvas. Students who remain engaged with the lectures and finish the homework assignments in a timely manner should not find the quizzes overly difficult.

The quizzes are intended to assess individual engagement with the course; students may not collaborate. However, use of course materials and external resources (including AI tools) is permitted.

The lowest two quiz scores will be dropped when calculating final grades. Furthermore, the quizzes are designed to take no more than 30 minutes, but one hour will be allowed. Together, these policies are intended to accommodate technical difficulties and occasional absences, whether due to illness or other reasons, as well as students receiving extra time (1.5x or 2.0x). Further accommodations will be rare but considered on a case-by-case basis. Please contact the instructors with any questions or concerns.

Homework [40%]

The homework expands on the lecture material and involves coding exercises. Some tasks are easy, with step-by-step instructions, while others require more initiative. They give students practical experience writing HPC code and working in cluster environment.

Collaboration between students on the homework is welcome, and the use of external resources (including AI tools) is encouraged. However, each student must submit their own work and is expected to understand every detail of their submissions.

Students have a budget of four late days, used to submit homework assignments late without penalty (up to four days in total throughout the semester). There are no fractional late days. Once a student has exhausted their late days, late homework submissions will be penalized by 25% per day of the total possible points. Further accommodations are unlikely, but will be considered on a case-by-case basis. If a student believes their grade for a homework question is incorrect, after comparing it with the solution, they may submit a regrade request via Gradescope one time.

Project [25%]

For the project, students will choose an algorithm or computational task and implement it using several of the parallel programming techniques introduced during the course.

Final Exam [25%]

The final exam will be administered in-person for CS-2050 students and via Proctorio for CSCI E-205 students. It will primarily consist of multiple choice and short answer questions.

The exam is fully "closed book," and no collaboration is permitted.

Compute Cluster

Our compute cluster is a shared, finite resource, meaning submitted job(s) might not execute immediately and could be delayed by several hours or more. For this reason, students are strongly encouraged to complete assignments as early as feasible, and extensions will not be granted simply because the cluster was busy leading up to a deadline.

Academic Integrity

The quiz, homework, project, and exam sections give specific expectations for those elements of the course. Students are further reminded of the Honor Code.

Accessibility Statement

Harvard University values inclusive excellence and providing equal educational opportunities for all students. Our goal is to remove barriers for disabled students related to inaccessible elements of instruction or design in this course. If reasonable accommodations are necessary to provide access, please contact the Disability Access Office (DAO). Accommodations do not alter fundamental requirements of the course and are not retroactive. Students should request accommodations as early as possible, since they may take time to implement. Students should notify DAO at any time during the semester if adjustments to their communicated accommodation plan are needed.

Students may also contact the instructors with accessibility questions or concerns.