In this lab, you will strengthen the "at-most-once" guarantees of your distributed Word Count application to achieve exactly-once semantics. You will implement a retry mechanism with exponential backoff on the client side and a duplicate detection mechanism on the server side to achieve linearizability.
gRPC runs on top of TCP, which reliably handles simple packet drops. However, in distributed systems, "reliability" is more complex.
Failures still happen: Servers can crash, overload, or the network partition can break the TCP socket.
The "Hang" Problem: If a server crashes while processing your request, your client might hang indefinitely waiting for a response that will never come.
The "Retry" Risk: If you simply retry a failed request, you risk executing the operation twice (e.g., incrementing a word count by 2 instead of 1).
To solve this, we need RPC Deadlines (to prevent hanging) and Linearizability (to prevent double-execution).
Please follow the Debugging Tools demo to finish installation of all necessary tools and get familiarized with tracing tools. OTel is required to complete this lab. DDB is highly recommended for understanding the behaviors of server and client together. (I used it a lot when I tried this lab).
Your code to Github classroom repo. Make yours: https://classroom.github.com/a/4r0SHAfZ
Your report under the main directory of your Github classroom repo. Use file name "report.pdf"
Your first task is to simulate a "partial failure" where the server processes the request but the client never receives the response. This creates an ambiguity: Did the request succeed?
Modify your client to set a short deadline (e.g., 100ms) for the AddWordCount RPC.
How to set a Deadline: gRPC uses ClientContext to control call settings. You must set an absolute deadline (not just a duration).
Include <chrono>.
Create a grpc::ClientContext.
Use context.set_deadline(std::chrono::system_clock::now() + std::chrono::milliseconds(500));.
Pass this context to your RPC stub.
If the server takes longer than 500ms, the call will return immediately with status.error_code() == grpc::DEADLINE_EXCEEDED.
Then, artificially induce a server-side delay or failure so the client times out.
Method 1: Sleep Injection (easier)
Alternatively, simply add a sleep() in your server handler that exceeds the client's deadline.
Method 2: Using DDB/GDB to Skip Response (advanced, optional)
We can use GDB to simulate a server crash after processing but before replying.
Run your server under DDB/GDB
Set a breakpoint at the line where the server sends the response (e.g., return Status::OK; or responder.Finish()).
Trigger an RPC from the client.
When the breakpoint hits, use the jump command to skip the response sending line.
GDB Command: jump <line_number_after_response>
VSCode with DDB: right click on the line to jump to and click "Jump to Cursor".
Tip: Attach to the server and add your breakpoint first, then resume the server before launching the client(or otherwise you may see a connection refused error).
Warning: jump just moves the instruction pointer. It does not clean up stack variables or run destructors, which simulates a hard crash effectively for this test.
Method Used: Describe which method you used to trigger the failure.
Observation: Include a screenshot of your OpenTelemetry traces (or log output) showing the RPC span ending with a DEADLINE_EXCEEDED status code, while the server logs show the operation completed.
Modify each update RPC methods to retry with exponential backoff.
Set a context.set_deadline. Start with 500 ms.
Loop with Exponential Backoff:
Call the RPC.
If Status::OK, return success.
If Status is retryable (e.g., UNAVAILABLE, DEADLINE_EXCEEDED), sleep for base_delay * 2^retries, then retry sending the exact same request object.
You may limit the number of retries to a reasonable number.
Observation: Set a long delay in the server to show the exponential backoff. Include a screenshot of your OpenTelemetry traces (or log output) showing the RPC spans ending with a DEADLINE_EXCEEDED status code, with increasing lengths.
Now we implement the "Reusable Infrastructure for Linearizability" (RIFL) concepts to handle these failures gracefully. To make it simple, we will not persist data to disk.
Every client instance must have a unique 64-bit ID.
In a real system, this might come from a coordination service (like ZooKeeper).
For this lab: You may randomly generate a uint64_t client ID at startup. Be careful on setting random seed correctly. You may search web or consult with ChatGPT on how to get good random number for the whole uint64_t range. I provided a sample code in rpcClient.cpp, inside the constructor of RpcClient. (you may reuse it)
You need two data structures to implement RIFL. One (rpcTracker) resides on the client-side to track all RPC IDs it dispatched to servers and to calculate the highest ACK IDs to confirm receipt of RPC responses. The other (unackedRpcResults) resides on the server-side to track the ongoing RPCs and finished but not-yet-acknowledged RPCs. unackedRpcResults also saves the RPC response from the original executions as well.
Get familiarize yourself with the two modules (rpcTracker.cpp/.h and unackedRpcResults.cpp/.h)
For the exception safety, UnackedRpcHandle class should be used in the RPC handler methods, instead of using uanckedRpcResults directly. It's like using lock_guard instead of using mutex directly.
Modify your accumulator.proto file to include RIFL metadata in all update RPC requests. "Empty" is updated already for you. ResetCounter RPC is half-implemented for reference.
Run ./build.sh or protoc to regenerate your stubs.
Modify each update RPC methods to include the RIFL metadata. For your reference, RpcClient::ResetCounter() is implemented already.
Obtain new sequence number (RPC ID) and acknowledgement number (ACK ID) from RpcTracker.
Set the corresponding fields in RPC requests.
When RPC reponse is received, report this to rpcTracker by invoking rpcFinished.
Modify each update RPC handlers to check the duplicate and avoid the duplicate execution by returning the saved results.
Use UnackedRpcHandle, instead of invoking UanckedRpcResults class directly.
UnackedRpcHandle::isDuplicate(), UnackedRpcHandle::isInProgress(), UnackedRpcHandle::savedResponse(), UnackedRpcHandle::recordCompletion(replyMsg) should be properly used. If you have questions, refer back to the header and source code.
AccumulatorServiceImpl::ResetCounter is already half-modified to help your understanding. But there are still two remaining tasks for ResetCounter.
TODO 1: rh.isInProgress() is not used in the current code. please put it at the right location to detect the in progress RPCs and
return Status(StatusCode::UNAVAILABLE, "RPC in progress, please retry later.");
TODO 2: The current code is directly saving "replyMsg", but this method cannot be used for other RPCs. Please change to saving the protobuf-serialized string of the RPC reponse messages. You may consult to ChatGPT to figure out how to serialize a protobuf message and get the std::string value.
Please finish implementation for all other RPCs.
Report for Milestone 3
Observation: Set a long delay in the server's RPC handler to trigger retry. Include a screenshot of your OpenTelemetry traces (or log output) showing the RPC spans ending with a DEADLINE_EXCEEDED status code and one with the saved response. Show all retries have the same RPC IDs.