AWS CloudWatch console

CloudWatch CLI

AWS CLI

CloudWatch API

AWS SDKs

Metric is the fundamental concept in CloudWatch.

Metrics are data about the performance of your systems.

Many AWS services provide free metrics for resources by default (such as Amazon EC2 instances, Amazon EBS volumes, and Amazon RDS DB instances).

You can also enable detailed monitoring for some resources, such as your Amazon EC2 instances, or publish your own application metrics. Amazon CloudWatch can load all the metrics in your account (both AWS resource metrics and application metrics that you provide) for search, graphing, and alarms.

Uniquely defined by a name, a namespace, and one or more dimensions

Represents a time-ordered set of data points published to CloudWatch.

Think of a metric as a variable to monitor, and the data points as representing the values of that variable over time. For example, the CPU usage of a particular EC2 instance is one metric provided by Amazon EC2. The data points themselves can come from any application or business activity from which you collect data.

Metrics are uniquely defined by a name, a namespace, and zero or more dimensions.

Each data point has a time stamp, and (optionally) a unit of measure

Data points can be either custom metrics or metrics from other
services in AWS.

Statistics can be retrieved about those data points as an ordered set of time-series data that occur within a specified time window.

When the statistics are requested, the returned data stream is identified by namespace, metric name, dimension, and (optionally) the unit.

Metrics exist only in the region in which they are created

CloudWatch stores the metric data for two weeks

Metrics cannot be deleted, but they automatically expire in 14 days if no new data is published to them.

NOTE: From Nov 2016 AWS provides Extended Metrics Retention

• One minute data points are available for 15 days.

• Five minute data points are available for 63 days.

• One hour data points are available for 455 days (15 months).

CloudWatch Logs allows you to monitor, store, and access your log files from sources including Amazon EC2 instances, Route 53, CloudTrail, and other AWS services.

For example, you could monitor logs from Amazon EC2 instances in real time. You could track the number of errors that have occurred in your application logs and send a notification if that rate exceeds a previously defined amount.

Amazon CloudWatch Events delivers a near real-time stream of system events that describe changes in AWS resources.

AWS resources can generate events when their state changes. For example, Amazon EC2 generates an event when the state of an EC2 instance changes from pending to running, and Amazon EC2 Auto Scaling generates events when it launches or terminates instances.

Using simple rules that you can quickly set up, you can match events and route them to one or more target functions or streams.

CloudWatch Events becomes aware of operational changes as they occur. CloudWatch Events responds to these operational changes and takes corrective action as necessary by sending messages to respond to the environment, activating functions, making changes, and capturing state information.

You can also use CloudWatch Events to schedule automated actions that self-trigger at certain times using cron or rate expressions.

A rule matches incoming events and routes them to targets for processing. A single rule can route to multiple targets, all of which are processed in parallel.

Rules are not processed in a particular order.

This enables different parts of an organization to look for and process the events that are of interest to them. A rule can customize the JSON message sent to the target by passing only certain parts or by overwriting it with a constant.

A target processes events.

Targets can include Amazon EC2 instances, AWS Lambda functions, Kinesis data streams, Amazon ECS tasks, Step Functions state machines, Amazon SNS topics, Amazon SQS queues, and built-in targets.

A target receives events in JSON format.

CloudWatch namespaces are containers for metrics.

A namespace allows you to categorize your metrics. If you are collecting a metric specific to a type of AWS resource, you can store the metric within the namespace for the type, such as AWS/EC2, or you can make up your own namespace, such as Customer/PurchaseData.

Metrics in different namespaces are isolated from each other, so that metrics from different applications are not mistakenly aggregated into the same statistics.

AWS namespaces all follow the convention AWS/<service>, for e.g. AWS/EC2 and AWS/ELB

Namespace names must be fewer than 256 characters in length.

There is no default namespace. Each data element put into CloudWatch must specify a namespace

Dimensions is a name/value pair that is part of the identity of a metric.

You can assign up to 10 dimensions to a metric.

Every metric has specific characteristics that describe it, and you can think of dimensions as categories for those characteristics.

Dimensions helps design a structure for the statistics plan.

Dimensions are part of the unique identifier for a metric, whenever a unique name pair is added to one of the metrics, a new metric is created.

Dimensions can be used to filter result sets that CloudWatch query returns

Every metric has specific characteristics that describe it, and you can think of dimensions as categories for those characteristics. Dimensions help you design a structure for your statistics plan. Because dimensions are part of the unique identifier for a metric, whenever you add a unique name/value pair to one of your metrics, you are creating a new variation of that metric.

Units represent the statistic’s unit of measure for e.g. count, bytes, % etc

Statistics are metric data aggregations over specified periods of time.

Aggregations are made using the namespace, metric name, dimensions, and the data point unit of measure, within the specified time period.

Period is the length of time associated with a specific statistic.

Each statistic represents an aggregation of the metrics data collected for a specified period of time.

Although periods are expressed in seconds, the minimum granularity for a period is one minute.

CloudWatch aggregates statistics according to the period length specified in calls to GetMetricStatistics.

Multiple data points can be published with the same or similar time stamps. CloudWatch aggregates them by period length when the statistics about those data points are requested.

Aggregated statistics are only available when using detailed monitoring.

Instances that use basic monitoring are not included in the aggregates

Metric data points that specify a unit of measure are aggregated separately. When you get statistics without specifying a unit, CloudWatch aggregates all data points of the same unit together.

CloudWatch does not aggregate data across regions.

CloudWatch aggregates statistics according to the period length that you specify when retrieving statistics.

For large datasets, you can insert a pre-aggregated dataset called a statistic set: MIN, MAX, SUM, AVG, Sample and Percentile.

Alarms can automatically initiate actions on behalf of the user, based on specified parameters.

Alarm watches a single metric over a specified time period, and performs one or more actions based on the value of the metric relative to a given threshold over a number of time periods.

Alarms invoke actions for sustained state changes only i.e. the state must have changed and been maintained for a specified number of periods.

Action can be a

• SNS notification

• Auto Scaling policies

• EC2 action – stop or terminate EC2 instances

After an alarm invokes an action due to a change in state, its subsequent behavior depends on the type of action associated with the alarm.

• For Auto Scaling policy notifications, the alarm continues to invoke the action for every period that the alarm remains in the new state.

• For SNS notifications, no additional actions are invoked.

An alarm has three possible states:

• OK—The metric is within the defined threshold

• ALARM—The metric is outside of the defined threshold

• INSUFFICIENT_DATA—Alarm has just started, the metric is not available, or not enough data is available for the metric to determine the alarm state

Alarms exist only in the region in which they are created.

Alarm actions must reside in the same region as the alarm

Alarm history is available for the last 14 days.

Alarm can be tested by setting it to any state using the SetAlarmState API (mon-set-alarm-state command). This temporary state change lasts only until the next alarm comparison occurs.

Alarms can be disabled and enabled using the DisableAlarmActions and EnableAlarmActions APIs (mon-disable-alarm-actions and mon-enable-alarm-actions commands).

Command Example:

• 1. >> aws cloudwatch put-metric-alarm --alarm-name NotesWriteCapacityUnitsLimit --metric-name ConsumedReadCapacityUnits --namespace AWS/DynamoDB --statistic Sum --period 60 --treat-missing-data missing --datapoints-to-alarm 5 --alarm-actions arn:aws:cloudwatch:us-east-2:111122223333:alarm:Notes-WriteCapacityUnitsLimit-BasicAlarm --dimensions "Name=InstanceId,Value=i-12345678"

CloudWatch does not aggregate data across regions. Therefore, metrics are completely separate between regions.

Metrics produced by AWS services are standard resolution by default.

When you publish a custom metric, you can define it as either standard resolution or high resolution.

When you publish a high-resolution metric, CloudWatch stores it with a resolution of 1 second, and you can read and retrieve it with a period of 1 second, 5 seconds, 10 seconds, 30 seconds, or any multiple of 60 seconds.

A log event is a record of some activity recorded by the application or resource being monitored.

Log event record contains two properties: the timestamp of when the event occurred, and the raw event message

Log Streams

A log stream is a sequence of log events that share the same source for e.g. logs events from an Apache access log on a specific host.

Log Groups

Log groups define groups of log streams that share the same retention, monitoring, and access control settings for e.g. Apache access logs from each host grouped through log streams into a single log group

Each log stream has to belong to one log group

There is no limit on the number of log streams that can belong to one log group.

Metric Filters

Metric filters can be used to extract metric observations from ingested events and transform them to data points in a CloudWatch metric.

Metric filters are assigned to log groups, and all of the filters assigned to a log group are applied to their log streams.

Retention Settings

Retention settings can be used to specify how long log events are kept in CloudWatch Logs.

Expired log events get deleted automatically.

Retention settings are assigned to log groups, and the retention assigned to a log group is applied to their log streams.

CloudWatch retains metric data as follows:

1. 1. Data points with a period of less than 60 seconds are available for 3 hours. These data points are high-resolution custom metrics.

   2. Data points with a period of 60 seconds (1 minute) are available for 15 days

   3. Data points with a period of 300 seconds (5 minute) are available for 63 days

   4. Data points with a period of 3600 seconds (1 hour) are available for 455 days (15 months)

Supported logging protocols

1. Apache log4net

2. Apache Log4j

3. Nlog

4. Serilog

Log Example:

1. {

2. "level": "Error",

3. "message": "Error processing notification",

4. "timestamp": "1591940416",

5. "context": {

6. "userId": “StudentA",

7. "type": "Lambda.Handler",

8. "env": "dev",

9. "component": "api",

10. "correlationId": "41e556-9e5-4c37-856e-3b623be",

11. "threadId": 16,

12. "member": "ProcessNotification",

13. "sourceFile": "D:\\a\\1\\s\\Lambda\\Handler.cs",

14. "exception": "<exception details>"

15. }

16. }

17. 
    

can help monitor applications and systems using log data

can help track number of errors for e.g. 404, 500, for even specific literal terms “NullReferenceException”, occurring in the applications, which can then be matched to a threshold to send notification

can be used to monitor particular API activity as captured by CloudTrail by creating alarms in CloudWatch and receive notifications

can help store the log data in highly durable storage, an alternative to S3.

log retention setting can be modified, so that any log events older than this setting are automatically deleted.

Can help log information about the DNS queries that Route 53 receives.

Subscriptions can help get access to real-time feed of logs events from CloudWatch logs and have it delivered to other services such as Kinesis stream, Kinesis Data Firehose stream, or AWS Lambda for custom processing, analysis, or loading to other systems

A subscription filter defines the filter pattern to use for filtering which log events get delivered to the AWS resource, as well as information about where to send matching log events to.

CloudWatch Logs log group can also be configured to stream data Elasticsearch Service cluster in near real-time

CloudWatch Logs allows searching and filtering the log data by creating one or more metric filters.

Metric filters define the terms and patterns to look for in log data as it is sent to CloudWatch Logs.

CloudWatch Logs uses these metric filters to turn log data into numerical CloudWatch metrics that can be put as graph or set an alarm on.