At the same time, the logs are written to the file sinks:

OTel Concepts

OTel is the state-of-the-art distributed tracing system, which is widely used in modern cloud.

The key feature of OTel toolset is producing a timeline of each RPC request, like the timeline generated by AddWordCount below. In production, a request can involve dozens of distributed services, and with OTel integration, developers can see a single timeline aggregated from many machines/processes. Therefore, OTel is the de facto standard for debugging and performance analysis for cloud applications, especially micro-services.

In the above timeline, every horizontal bar is called a span, usually representing the lifespan of a function, an RPC, or just a segment of code. Notably, the longer green span is the parent span of the shorter green span, as the short one is a "subcomponent" of its parent.

More specifically, the span wc_server /Accumulator/AddWordCount is automatically generated by OTel's gRPC interceptor. For the demo and Raft labs, when compile with tracing enabled, the interceptor will automatically intercept all gRPC calls, recording the request and response contents as span attributes (shown below).

On contrary, the AddWordCountManualSpan is manually created (link to code). Manually created spans are usually used for tracing function calls inside a server, giving you a magnified view of the RPC handling process. Unlike intercepted spans, manual spans' attributes need to be manually set as well.

OTel Toolset Detailed Usage

0. Install OTel

In both the demo repo and the Raft lab repo, Ubuntu VM can use the install_otel.sh directly.

1. Building your binaries with tracing enabled

$ cd ~/CSCI-546-debugging-tools-demo/ # or the Raft lab

$ mkdir build

$ cmake .. -DTRACING=ON

$ make -j$(nproc)

2. Launch the Jaeger/Grafana collector + web UI server

Both Jaeger and Grafana can visualize the tracing spans, but they are different:

• Jaeger UI is very simple and intuitive. It's user-friendly for OTel beginners. However, it doesn't support logs attaching to spans (introduced later).

• Grafana UI is much more complex, which contains Grafana Tempo for tracing, and Grafana Loki for logging. In the above example, you can see Grafana has a button "Logs for this span" to view logs attached each span.

Under the tools/ directory, we packaged docker-compose for both Jaeger and Grafana so that you can launch either of them in one line:

$ cd tools/jaeger-suite/

$ docker compose up -d

OR

$ cd tools/grafana-suite/

$ docker compose up -d

Note: you should NOT launch both at the same time. To remove the deployment, switch to corresponding directory, and do

$ docker compose down

# OR

$ docker compose stop # temporary stop, doesn't delete previous trace data

3. Port forwarding and accessing UI in browser

By default, Jaeger uses port 16686 and Grafana uses port 3000 to expose their web UI. If you are using a VM, the port is exposed only inside the VM, so you need to forward it to your host machine before connecting to it in your browser. You can do this by ssh port forwarding, OR a simpler way is the "Add Port" button in VS Code's "Ports" menu:

Then, you can use your browswer to connect to localhost:16686 or localhost:3000, and Hooray! You are ready to launch some word count requests/raft nodes and then start exploring the UI! Before you start, review the notes in "Demo Repo" section above.

OTel + spdlog + Grafana UI = Gluing Logs with Spans

When you have a huge volume of logs (which is very normal in distributed systems), reviewing logs become difficult. A powerful feature of this toolset is allowing you to click on a span in the timeline, and see the logs associated with it. So you can walk through the logs inside a visual timeline.

When you compile labs with cmake .. -DTRACING=on , the spdlog will writes to OTel sink, and then logs are fed into Grafana. If you use Jaeger, logs are dropped. Then, when you view a trace in Grafana Tempo, you can click on the "Logs for this span" button to view logs inside that span. Note that you need to manually create spans to associate with logs, because spdlog doesn't attach to gRPC intercepted spans automatically. If a log is not associated with a span, you can still view it in Grafana Loki directly.

For more details on how to glue logs with spans, see this code snippet.

Since VSCode is the most widely used IDE, we'll use it for interactive debugging.

To get started:

1. Open the Debug panel by clicking the Run and Debug icon in the sidebar (or press Ctrl+Shift+D / Cmd+Shift+D).

2. Configure your debugging settings by either clicking the gear icon or directly opening .vscode/launch.json.

Note that debugging requires both a language extension and a compatible debugger. For C++ development:

• GDB: Install the C/C++ extension (ms-vscode.cpptools), which includes GDB support

• DDB: Install the DDB extension along with the DDB debugger for distributed debugging

Refer to the following sections for installation instructions.

Tips

1. How to search in Call Stack(when we are trying to find another thread based on threadId, similar to what we are doing in class) ?

To search within the call stack:

1. Open keyboard shortcuts settings and find the keybinding for list.find

2. Click on any thread or frame in the Call Stack panel to focus it

3. Press the keybinding to open the search box within the panel

Step 1. launch ddb process

Start the debugger from the VSCode Debug panel.  (For details on each configuration option, see  Configure VSCode | DDB)

Step 2. connect your process to ddb 

Usually at the start of main entrypoint through ddb connecter (if you are using demo code or the course's lab don't worry, we've added it for you, just use the --ddb flag)

if (args.enable_ddb) {

     auto cfg = DDB::Config::get_default("127.0.0.1")

                   .with_alias(proc_alias)

                   .with_logical_group(proc_alias);

    cfg.wait_for_attach = args.wait_for_attach;

    auto connector = DDB::DDBConnector(cfg);

    connector.init();

}

Step 3. launch with ddb_runapp wrapper

ddb_runapp ./wcServer --bind 127.0.0.1:50051 --ddb

In the Call Stack panel, the server's processes will appear but won't be running yet—they're paused when attached to the debugger. You can either click the Continue button to let the server run, or add breakpoints first. (Adding breakpoints while the server is running works fine too.)

Step 4. Add a breakpoint to the code

Try to click at the left side of a line of code, a popup will appear, prompting you to select which session(s) to add a breakpoint to. In this case, we only have one session, but if you launch multiple instances, you'll see multiple sessions listed.

After you click OK, a breakpoint should be added shortly. If "processing" displays for too long, try adding it again. Then click Continue to let the server run.

Step 5. Run client and inspect server breakpoint

After you see the server is running, launch client

ddb_runapp ./wcClient --dest 127.0.0.1:50051 --text "Hello!" -ddb

After this step, remember to click Continue. Processes are interrupted when added to DDB.

The breakpoint should be triggered on the server side. DDB has a powerful feature: it can perform stack backtraces across process boundaries. The red rectangle on the left marks the boundary between two processes—below it is the client stack, and above it is the server stack. This allows you to inspect the entire call stack at once.