CloudLab Setup for DDB

For easy access to DDB (interactive distributed debugger), we provide a pre-configured Cloudlab setup. Enrolled students may access CloudLab, which provides bare metal servers for free.

Join CloudLab Project

First, create an account: https://www.cloudlab.us/
- Use your roster name and USC email so that we can cross-check if you have signed up. If you haven't signed up but want to participate, please fill out the signup form before registering the account.
Then, request to join: https://www.cloudlab.us/signup.php
- Select "join existing project", and type in "uscds"
- Course staff will approve you after review.

This resource is primarily for research. So, let's stay away from the most recent powerful nodes. For our coursework, d710 machine type should be powerful enough. Let's stay on this.

Spin Up Cluster with Pre-configured Experiment profile

We provide a CloudLab profile to help you instantiate a cluster of nodes with an attached NFS server and DDB dependencies. You can store your source file in the NFS server and compile your code directly into the NFS server. This can be handy when deploying multiple instances (the same binary) across the cluster.

You can access the profile at this link: https://www.cloudlab.us/p/c3e4d2b7d048ddbd395c12bcabfcffdd7afa528c

Instantiate Cluster from the Profile

First tab "Select a profile", click Next.
Second tab "Parameterize", change the Number of Nodes as you need. You can leave the other field as the default value and click Next.
Third tab "Finalize", fill out the Name field with any preferred name. Ensure select "uscds" for the Project field, and "Emulab" for the Cluster field. An example screenshot is attached below.
Fourth tab "Schedule", leave all fields as default and click Finish.
After successfully instantiating the cluster, don't forget to extend the experiment, as the default initial period is only 16 hours.

Note: It may take a while for the CloudLab to fully instantiate the cluster. If it fails to provision, please retry after a while.

This profile will create a cluster for you. All nodes are marked with the name node* (* is the incremental id). You won't need to access the "nfs" node, which is merely used as an NFS server. All nodes have pre-installed DDB dependencies.

All nodes will have a shared NFS folder mounted at /nfs file path. You can store files in this folder. This NFS folder is shared across all nodes at the same mount point.

Prepare your Project on CloudLab

Check if the cluster is ready

When the cluster is ready, you will see the State: ready. Please do not proceed if it is not fully ready. Besides, for all nodes, you should see both green for Status (ready) and Startup (Finished).

Set up your Project Files

Let's first log into node0 and set up your lab files on that machine. In my setup, I have two experiment nodes and one nfs node. In my case, I need to ssh ybyan@pc426.emulab.net. Please change the hostname and username based on your case accordingly.

After ssh-ed into this machine, let's clone your lab project into your home folder.

$ cd ~

$ git clone YOUR-PROJECT-REPO-URL

Note: we pre-install gh (GitHub CLI tool) for you. If you don't want to manually set up the ssh key or token with GitHub. You can run `gh auth login` to authenticate with GitHub. Afterward, you can git clone without specifying the token.

Set up DDB-related Files

We have prepared all DDB-related files in the folder /opt/ddb_configs. It includes these files/folders:

-rw-r--r-- 1 ybyan flashburst 2179 Oct 15 03:23 0001-adapt-latest-ddb.patch

drwxr-xr-x 2 ybyan flashburst 4096 Oct 14 23:27 .ddb

-rw-r--r-- 1 ybyan flashburst 1919808 Oct 15 03:23 ddb-debugger-0.0.1.vsix

drwxr-xr-x 2 ybyan flashburst 4096 Oct 14 23:27 .vscode

For a quick setup, you can run cp -r /opt/ddb_configs/. <YOUR_PROJECT_PATH>. However, please do note that if you have your own .vscode configuration files, they might get overwritten. Please copy that with caution.

Prepare your VS Code

Setup SSH in VSCode for CloudLab Connection

This part is very similar to the Multipass + VS Code setup here. Please follow the guide and adjust it accordingly. After this, let's open up the SSH connection and open the folder of your project. You probably have a view similar to this:

Install DDB VSCode Extension

You can find the extension package file named ddb-debugger-0.0.1.vsix. Right click on the file in the file explorer and click on Install Extension VSIX in the popup view.

If everything goes well, you will see a popup panel indicating "Completed installing extension."

Patch Lab Codes with DDB Support

To integrate DDB into your project, you need some minor changes to the codebase. No worries; this process won't touch your implementation code. It only changes the skeleton code.

To apply the patch, please run:

git apply 0001-adapt-latest-ddb.patch

If you modified some skeleton code or build files, you may need to resolve the conflict. If you have any questions, please post on the Piazza. If you wonder what kind of patch it performs, please refer to the appendix.

Then, you are all set!

Build your Project

Now, it's time to re-build your lab on the new CloudLab cluster.

Before building, please don't forget to run ./setup.sh . The building procedure is the same as the original lab spec.

Note: To enable a full debugging experience, when configuring cmake, please run cmake -DCMAKE_BUILD_TYPE=Debug .. (-CMAKE_BUILD_TYPE=Debug flag will enable full debug symbol).

Final Check

Let's check if the DDB is now running correctly. Please be sure to build your project first.

Now, let's open up the vscode debug panel, and you will see a debug task named "Launch DDB" has already been created for you. Image is shown as below:

After clicking the run (green triangular) button, the debug control panel will pop up. Please wait until it's ready. The image is shown below:

Now, let's run the multinode app for a test run. After running the multinode app, you will see the "Call Stack" panel is populated with thread information, and the debug control panel also shows a continue button instead of the pause button. The image is shown below:

You are good to go if you can see the call stack filled up!

Appendix

What are things the patch changes?

It uses a pre-built and pre-installed hacked version of gRPC.
It modifies the cmake file in the proto folder to properly link to an external library DDB needs.
It modifies the node app, so that the node app properly initializes DDB.

Additional Resources

Page updated

Report abuse