This document provides suggestions based on previous mentor’s comments and what we have seen to be good practice before, during, and after a hackathon. These guidelines outline a strongly recommended practice when serving as a mentor at a GPU hackathon. Of course, each team brings nuances which may require some modification to these recommendations. This document is always evolving as we continue to build a deeper understanding of how to approach guiding developers to success at a one-week hackathon.
If, at any time, you feel something is missing or needs to be adjusted in this document, feel free to reach out to email@example.com with your suggestions
Identify the compute platform you will use at the hackathon
At the hackathons, there is often access to a variety of compute resources (e.g. cloud platform, organizer/partner systems, or your team’s in-house system). Teams that have been successful have made sure that everyone on their team, including the mentors, have access to the same compute resources. Whenever possible, it is important that you and your team access the compute platform before the hackathon. This gives you and your team the opportunity to understand the environment setup and job scheduler on that system. Learning how to move around on the chosen compute system before the hackathon helps maximize the time your team will spend on coding. Be aware that at some hackathons, such as those organized by OLCF, this may not be possible due to restrictions on Department of Energy Systems. Always consult with the organizers to determine compute platform availability before an event.
Your team’s application should compile, run, and give correct answers
During the application process, the organizers try to find applications that are ready for a GPU hackathon. However, it is possible that the application may not be completely ready to handle the API you have in mind for them. It is important to make sure that your team’s code compiles with the compiler required to implement the API. For example, if you and your team are planning to use OpenACC, you should make sure your team’s source code can be compiled with the PGI compilers or GCC-8.
Additionally, it is possible that your team will be running their application on a computing platform that is new to them. Make sure that the code compiles with the appropriate compilers on the planned compute platforms. The generated executable should be verified before coming to the hackathon (whenever possible). Help guide your team in the right direction to make sure their code is ready for a coding sprint.
Your team’s application should be self contained
Ideally, the focus of the 5 day hackathon should be limited to a few thousand lines of code. System dependencies (e.g. Cray computing environment, hard-coded paths in build system, etc) should be removed. If there are external dependencies (e.g. NetCDF, HDF5, BLAS/LAPACK) that are required, it is imperative that these dependencies, along with the required compilers, are communicated to the organizers and system administrator well before the event. At some events, this communication happens during the registration process.
Work with your team to identify external dependencies and communicate these requirements to the organizers and system administrator.
Your team’s code should be configured to run in a short amount of time
During the 5 day coding sprint, you and your team will be making modifications to the code base and running the code to ensure correctness. Successful teams have found that many small changes, accompanied with frequent code execution and output verification, is a desirable workflow. To fit this workflow and be productive at the hackathon, it is necessary that the test case used to verify/validate the code, runs in a short amount of time; this permits rapid feedback that guides changes you will be making in the code.
Your team’s code should run on one process
If your team’s application is being ported to the GPU for the first time or if you are working on kernel optimization (and the problem size is small enough to fit on the target GPU) it’s recommended that your team’s code and example test case run in serial or only on a single GPU. Working with MPI in addition to a GPU programming API can result in more hurdles to overcome that just a single GPU application.
Even if the end goal is to run on multiple GPUs, if there is no GPU offloading in the code yet, it is highly recommended that the initial focus be on a single GPU application. This is more than enough work for a single hackathon. Optimization, such as setting up GPU-to-GPU communication and tuning for the target architecture can take up a hackathon itself, so unless this is the focus of your time at the event, it is recommended that you target a single GPU.
There should be a method for automatically verifying the code’s output
Although the primary goal for many teams is to have their code run faster on GPUs, the ported or tuned application should be producing the correct results. Because of this, your team should have a fairly automatic way of robustly verifying the correctness. Integrated metrics, such as RMS values, can hide changes in the produced results. Ideally, binary files that store the program’s state (e.g. the velocity, temperature, and pressure in a fluids code) should be compared against a reference set using “diff” or some hash function comparison.
When actively developing/debugging, it’s a good idea to generate the reference data and the “modified code” data using floating-point safe compiler flags. Only use less safe optimizations when benchmarking. In the event bit-for-bit reproducibility cannot be achieved relative to reference output, it’s good to have an application (written by your team) to compare two sets of application output.
Generate a call graph of the code before the hackathon
As someone who did not develop your team’s code, it is useful to have a call graph on hand that highlights hot-spots in the code and depicts the relationship between subroutines/functions. The least expensive way to do this is using valgrind’s callgrind tool and kcachegrind to visualize the callgraph.
Below are the steps for doing this on a linux system. Note that valgrind and kcachegrind need to be installed.
- Compile the code, from scratch, with the -g compiler flag enabled.
- Run the code with valgrind --tool=callgrind. For example
valgrind --tool=callgrind ./my_exe
- Running your application with callgrind generates a file, callgrind.XXXX.out, where the XXXX are replaced with the process id (assigned by the OS) for the application. This file can be visualized calling kcachegrind from the command line.
- Explore the GUI window and find the tab labeled ‘call graph’. Once you have the call graph in view, you can right click and export the call graph to a png.
Generate a profile of the code before the hackathon
To accompany the call graph, your team should generate a profile of their application before coming to the hackathon. This can be done with any profiler of your choice. You should be aware that some teams have not used a profiler before and you may need to suggest a profiler to use and provide them with some documentation to get started. In addition to speeding up code, the team’s are there to learn from the experts (you) the processes for porting to new hardware and performance tuning.
The goal is to identify which subroutines/functions you will be targeting first. Tackling the more expensive routines early on will likely give the largest speedup. This introduces high morale on your team early in the week and can be the encouragement needed for your team to want to continue this work on their own.
During the Hackathon
On the first day of the hackathon, in the morning, a member of your team will present an overview of the application and the goals for the week. The presentation should have 3-5 slides and take no more than 5 minutes to present. Your team should communicate
- Who they are
- What the application does
- The algorithm(s) that occupy most of the runtime currently
- The target for the week
The goal of the initial presentation is to present your plan clearly to other attendees, organizers, and mentors to get feedback on the approach. This helps make sure your team is on the path to success!
The presentation should be designed to communicate to other attendees at the hackathon. Typically, there is a wide variety of scientific domains in addition to computer scientists at the hackathon. Showing equations and explaining the impacts of the science does not usually help achieve the goal of these initial presentations and is therefore not recommended.
As a mentor, make sure the goals of the initial presentation are communicated to your team. Help them develop a short and succinct presentation to get feedback from other attendees.
Daily Scrum Sessions (Stand-ups)
Every morning, each team will give a short (2-3 minute) update expressing
- Where they are now
- Where they are going
- Where they are struggling
The goal of the daily scrum sessions is to check in with everyone and obtain additional feedback from other groups. We have found that, despite domain science differences, teams often come across the same hurdles are willing to share their solutions with others. Typically, we have asked mentors to deliver the updates at the first scrum session.
The presentations for the scrum sessions should include profiles and code snippets. Ideally, each team would show routine speedups/slowdowns for the routines they are actively working on. At these presentations, no equations should be shown and there is no need to reiterate the scientific background of the application. The goal is to let everyone know where your team is and to seek input to help get past hurdles.
Publications and Recognition
We recognize that you may be volunteering your time to help out at a hackathon. It is encouraged that you push your team towards publishing on their work. Becoming a mentor is a good mechanism for making new connections and becoming a contributing author on scientific publications. Additionally, you can ask your teams, at the very least, to recognize your contributions and assistance in the code development in any of their future publications with the code you worked on. Asking them to include a statement like
“We’d like to thank <Mentor Name> for their contributions towards porting and accelerating <Software Name> for use on < target compute system >. This work was accomplished thanks their expertise and patience at the <Hackathon name>.”
in the acknowledgements section of a publication is usually well received and helps recognize your contributions to the community.