Post date: Feb 26, 2019 2:43:15 PM

Amazon Kinesis Data Streams to collect and process large streams of data records in real time

The following are typical scenarios for using Kinesis Data Streams:

• Accelerated log and data feed intake and processing

• Real-time metrics and reporting

• Real-time data analytics

• Complex stream processing

Kinesis Data Streams Key Concepts

Kinesis Data Stream

A Kinesis data stream is a set of shards. Each shard has a sequence of data records. Each data record has a sequence number that is assigned by Kinesis Data Streams.

 Kinesis Data Streams Limits

• • There is no upper limit on the number of shards you can have in a stream or account. It is common for a workload to have thousands of shards in a single stream.

  • There is no upper limit on the number of streams you can have in an account.

  • A single shard can ingest up to 1 MiB of data per second (including partition keys) or 1,000 records per second for writes. Similarly, if you scale your stream to 5,000 shards, the stream can ingest up to 5 GiB per second or 5 million records per second. If you need more ingest capacity, you can easily scale up the number of shards in the stream using the AWS Management Console or the UpdateShardCount API.

  • The default shard limit is 500 shards for the following AWS Regions: US East (N. Virginia), US West (Oregon), and EU (Ireland). For all other Regions, the default shard limit is 200 shards.

  • The maximum size of the data payload of a record before base64-encoding is up to 1 MiB.

  • GetRecords can retrieve up to 10 MiB of data per call from a single shard, and up to 10,000 records per call. Each call to GetRecords is counted as one read transaction.

  • Each shard can support up to five read transactions per second. Each read transaction can provide up to 10,000 records with an upper limit of 10 MiB per transaction.

  • Each shard can support up to a maximum total data read rate of 2 MiB per second viaGetRecords. If a call to GetRecords returns 10 MiB, subsequent calls made within the next 5 seconds throw an exception.