Amazon Kinesis Data Streams

Overview

Kinesis Data Streams collects and stores streaming data in real time.

Real-time data is created and used on the spot: website click streams, internet-connected devices (a connected bicycle), and metrics and logs from servers.

• Producers send data in: your own application code (SDK), or a Kinesis Agent installed on servers for metrics and logs.
• Consumers process it in real time: your own application code, Lambda, Amazon Data Firehose (10-Amazon-Data-Firehose), or analytics with Managed Service for Apache Flink.

Features

Capacity Modes

A shard is basically how big your stream is. More shards means more throughput. Example: 10,000 records/s or 10 MB/s needs 10 shards.

Hands-On Walkthrough

Create the stream

1. Open Kinesis. Options: Data Streams, Data Firehose, Data Analytics (now Amazon Managed Service for Apache Flink). Choose Data Streams.
2. Pricing shown: $0.05 per hour per shard, plus a cost per PUT. There is no free tier, so you pay while the stream exists. Skip the demo if you do not want to spend money, and delete the stream quickly.
3. Name it DemoStream.
4. Capacity mode:
   • On-demand: max throughput 10 GB/s write and 20 GB/s read in us-east-1, us-west-2 and eu-west-1 (200 MB/s write and 400 MB/s read in other Regions, as in the course) with enhanced fan-out. Pay-per-throughput, no free tier.
   • Provisioned: use the shard estimator (records per second, record size, number of consumers). Choose 1 shard (1 MB/s write, 2 MB/s read, cheapest). 10 shards multiplies everything by 10.
5. Click Create data stream.

Console tour

• Producers recommended: Kinesis Agents, SDK (low-level producers), KPL (high-level, better API). All on GitHub.
• Consumers: Amazon Managed Service for Apache Flink (formerly Kinesis Data Analytics), Kinesis Data Firehose, Kinesis Client Library, and Lambda (not shown as an option).
• Monitoring shows records sent. Configuration lets you scale shards (for example 1 to 5), add tags and configure enhanced fan-out.

Produce and consume with CloudShell

1. Open CloudShell (icon next to the bell). It is free and inherits your IAM credentials and the launch region (us-east-1). Check the CLI version with aws --version. CloudShell has version 2, so use the v2 commands (the demo shows 2.1.16, as spoken).
2. Send a record with aws kinesis put-record: --stream-name DemoStream, --partition-key user1, --data "user signup", and --cli-binary-format raw-in-base64-out (needed for text data). Data with the same partition key goes to the same shard (only one here).
3. The response shows shardId-0000… (the first shard) and a sequence number. Repeat with user login and user logout. CloudWatch metrics (PutRecord) take a while to show.
4. Consume: run describe-stream to find the shard (shardId-0000… (the first shard)). With the CLI or low-level SDK you must specify the shard; the KCL handles this for you.
5. Request a ShardIterator with --shard-iterator-type TRIM-HORIZON (as spoken in the transcript), which reads from the very beginning of the stream. The other option reads only records from that moment onwards..
6. Run kinesis get-records --shard-iterator <value>. Records show PartitionKey user1, timestamps and base64-encoded data. Decoding with a base64 website gives the original text (user signup, user login).
7. The response includes a NextShardIterator, which you use to continue where you stopped. Your code must iterate over it.

This low-level flow is the shared consumption mode (not enhanced fan-out). Keep the stream open for 10-Amazon-Data-Firehose.

Questions & Answers