AWS X-Ray helps developers analyze and debug production, distributed applications for e.g. built using a microservices lambda architecture

X-Ray provides an end-to-end view of requests as they travel through the application, and shows a map of the application’s underlying components.

X-Ray helps to understand how the application and its underlying services are performing to identify and troubleshoot the root cause of performance issues and errors.

X-Ray to analyze both applications in development and in production, from simple three-tier applications to complex microservices applications consisting of thousands of services.

X-Ray can be used with distributed applications of any size to trace and debug both synchronous requests and asynchronous events.

X-Ray can be used to track requests flowing through applications or services across multiple regions. X-Ray data is stored locally to the processed region and customers can build a solution over it to combined the data.

Trace data sent to X-Ray is generally available for retrieval and filtering within 30 seconds of it being received by the service.

X-Ray stores trace data for the last 30 days. This enables you to query trace data going back 30 days.

Integration

1. X-Ray integrates with applications running on EC2, ECS, Lambda, and Elastic Beanstalk.

2. X-Ray SDK automatically captures metadata for API calls made to AWS services using the AWS SDK.

3. X-Ray SDK provides add-ons for MySQL and PostgreSQL drivers.

4. For Elastic Beanstalk, include the language-specific X-Ray libraries in your application code.

5. Applications running on other AWS services, such as EC2 or ECS, install the X-Ray agent and instrument the application code.

Simple setup: Instrument your application with the X-Ray SDK and install the X-Ray daemon. 

1. The daemon is already installed for Lambda and Elastic Beanstalk environments.

Trace

1. An X-Ray trace is a set of data points that share the same trace ID.

2. Trace helps track the request, which is assigned a unique trace id, while it navigates through services

3. Piece of information relayed by each service in the application to X-Ray is a segment, and a trace is a collection of segments.

4. A trace collects all the segments generated by a single request.

Segment

1. An X-Ray segment encapsulates all the data points for a single component of the distributed application for e.g. authorization component.

2. The compute resources running your application logic send data about their work as segments. 

3. Segments include system-defined and user-defined data in the form of annotations and are composed of one or more sub-segments that represent remote calls made from the service.  for e.g. database call and its result within the overall request/response.

4. Elements:

   1. Host – Hostname, alias, or IP address

   2. Request – Method, client address, path, user agent

   3. Response – Status, content

   4. Work done – Start and end times, subsegments

   5. Issues that occur – Errors, faults and exceptions, including automatic capture of exception stacks

Subsegments

1. A segment can break down the data about the work done into subsegments. 

2. Subsegments provide more granular timing information and details about downstream calls that your application made to fulfill the original request. 

Annotation

1. An X-Ray annotation is system-defined or user-defined data.

2. Annotation is associated with a segment and a segment can contain multiple annotations.

3. System-defined annotations include data added to the segment by AWS services

4. User-defined annotations are metadata added to a segment by a developer

Errors

1. X-Ray errors are system annotations associated with a segment for a call that results in an error response.

2. Error includes the error message, stack trace, and any additional information for e.g, version to associate the error with a source file.

Metadata

1. Key-value pairs with values of any type, including objects and lists, but that are not indexed. 

2. Use metadata to record data that you want to store in the trace but don't need to use for searching traces.

Sampling

1. X-Ray collects data for significant number of requests, instead of each request sent to an application, for performant and cost-effectiveness.

2. X-Ray should not be used as an audit or compliance tool because it does not guarantee data completeness.

3. By default, the X-Ray SDK records the first request each second, and five percent of any additional requests. 

   1. One request per second is the reservoir. 

   2. Five percent is the rate at which additional requests beyond the reservoir size are sampled. 

4. Customizing the sampling rules

   1. Rule name (string) – A unique name for the rule.

   2. Priority (integer between 1 and 9999) – The priority of the sampling rule. Services evaluate rules in ascending order of priority, and make a sampling decision with the first rule that matches.

   3. Reservoir (non-negative integer) – A fixed number of matching requests to instrument per second, before applying the fixed rate. The reservoir is not used directly by services, but applies to all services using the rule collectively.

   4. Rate (number between 0 and 100) – The percentage of matching requests to instrument, after the reservoir is exhausted. The rate can be an integer or a float.

   5. Service name (string) – The name of the instrumented service, as it appears in the service map.

      1. X-Ray SDK – The service name that you configure on the recorder.

      2. Amazon API Gateway – api-name/stage.

   6. Service type (string) – The service type, as it appears in the service map. For the X-Ray SDK, set the service type by applying the appropriate plugin:

      1. AWS::ElasticBeanstalk::Environment – An AWS Elastic Beanstalk environment (plugin).

      2. AWS::EC2::Instance – An Amazon EC2 instance (plugin).

      3. AWS::ECS::Container – An Amazon ECS container (plugin).

      4. AWS::APIGateway::Stage – An Amazon API Gateway stage.

      5. AWS::AppSync::GraphQLAPI – An AWS AppSync API request.

   7. Host (string) – The hostname from the HTTP host header.

   8. HTTP method (string) – The method of the HTTP request.

   9. URL path (string) – The URL path of the request.

      1. X-Ray SDK – The path portion of the HTTP request URL.

      2. Amazon API Gateway – Not supported.

   10. Resource ARN (string) – The ARN of the AWS resource running the service.

       1. X-Ray SDK – Not supported. The SDK can only use rules with Resource ARN set to *.

       2. Amazon API Gateway – The stage ARN.

   11. (Optional) Attributes (key and value) – Segment attributes that are known when the sampling decision is made.

       1. X-Ray SDK – Not supported. The SDK ignores rules that specify attributes.

       2. Amazon API Gateway – Headers from the original HTTP request.

5. Example:

   1. Let us suppose that we specify a value for Reservoir Rate as 60 and Fixed Rate as 20. Now, if your application receives 200 requests in a second, then the total number of requests that would be traced or sampled will be: - 

   2. "Reservoir Rate" + Fixed Rate % [(Total Requests - Reservoir Rate)] 

   3. - 60 + (200-60) * 20%

   4. - 60 + (140) * 20%

   5. - 60 + 28 

   6. - 88 

X-Ray agent

1. X-Ray agent helps collect data from log files and sends them to the X-Ray service for aggregation, analysis, and storage.

2. Agent makes it easier for you to send data to the X-Ray service, instead of using the APIs directly, and is

3. Agent is available for Amazon Linux AMI, Red Hat Enterprise Linux (RHEL), and Windows Server 2012 R2 or later operating systems.

Configuring the AWS X-Ray daemon

1. ./xray -n us-east-2: To get started, the only option that you need to know is -n or --region, which you use to set the region that the daemon uses to send trace data to X-Ray. 

2. ../xray -r : Assume an IAM role while saving results in different accounts.

3. ./xray -t : Bind a different TCP port for the X-Ray service

4. ./xray -b : Bind a different UDP port for the X-Ray service

5. ./xray -o : Can be used while running X-Ray daemon locally & not on Amazon EC2 instance. This will skip checking Amazon EC2 instance metadata. Test locally

6. 
   

Using X-Ray with Lambda