AWS -Amazon Kinesis

Kinesis is a cloud-based managed alternative to Kafka. Kinesis is a system used for building real-time data pipelines and streaming apps and storing the same data to AWS Redshift or S3.

Ingest and process streaming data with Kinesis streams

Amazon Kinesis Data Streams to collect and process large streams of data records in real time

Accelerated log and data feed intake and processing

Real-time metrics and reporting

Real-time data analytics

Process data with your own applications, or using AWS managed services like Amazon Kinesis Data Firehose, Amazon Kinesis Data Analytics, or AWS Lambda.

Replace existing Nifi enrichment and transformation pipeline

Provide targeted and directed data pipelines

Deliver streaming data with Kinesis Firehose delivery streams

Continuously collect, transform, and load streaming data into destinations such as Amazon S3 and Amazon Redshift.

Replace a complex flow on Nifi to bundle and compress the data

    Manually bundled and compressed data

 Makes the data easy to use for ATHENA

     Partition support

Lesson Learned

    JSON to Parquet Transforms 

Analyze streaming data with Kinesis analytics applications

Run continuous analysis on streaming data from Kinesis data streams and Kinesis Firehose delivery streams.

Filter data streams

Aggregate data

Anomaly and outlier detection

Ingest and process media streams with Kinesis video streams

Build applications to process or analyze streaming media

http://github.com/awslabs/aws-fluent-plugin-kinesis

http://github.com/awslabs/kinesis-log4j-appender

http://github.com/awslabs/amazon-kinesis-scaling-utils