Amazon Data Firehose

Overview

Amazon Data Firehose sends data from sources into target destinations. It used to be called Kinesis Data Firehose and was renamed because it does far more than Kinesis. It is fully managed, automatically scales, is serverless and you pay only for what you use.

Flow

1. Sources / producers: your applications and clients (SDK), the Kinesis Agent, and services Firehose can pull from, such as Kinesis Data Streams, CloudWatch Logs and Events, and AWS IoT (see 09-Amazon-Kinesis-Data-Streams).
2. Optional transformation with a Lambda function (for example CSV to JSON before S3).
3. Records accumulate in the buffer and are flushed in batches.
4. Destinations:
   • AWS: Amazon S3, Amazon Redshift (analytics), Amazon OpenSearch. Remember these three.
   • Third-party partners: Datadog, Splunk, New Relic, MongoDB.
   • Anything else: a custom HTTP endpoint.
5. Optionally write all data or only failed data to an S3 bucket for backup.

Data Formats

• Input: CSV, JSON, Parquet, Avro, text, binary.
• Conversion built in: to Parquet or ORC. Compression with gzip or snappy.
• Custom conversion or transformation: AWS Lambda.

Kinesis Data Streams vs Amazon Data Firehose

Hands-On Walkthrough

1. In Kinesis, open Delivery streams and create one. The diagram shows: ingest from producers (a Kinesis Data Stream as in this demo, direct PUTs, Kinesis Agents, CloudWatch, IoT Core, EventBridge, your own SDK apps), optional Lambda transform, then load to targets (S3, OpenSearch Service, Redshift, Apache Iceberg Tables, third-party and custom HTTP endpoints).
2. Source: Kinesis Data Streams and pick DemoStream (from the previous demo). Destination: Amazon S3. The delivery stream name is generated automatically.
3. Transform and convert records (optional): Lambda can transform, filter, uncompress, convert and process records. Convert record format (Parquet or ORC) is available but out of scope here.
4. Destination: choose or create an S3 bucket. Dynamic partitioning: no. S3 bucket prefix and error output prefix: not set.
5. Buffer hints:
   • Buffer size: Firehose writes 5 MB to the buffer by default before delivering. A larger size is more efficient (the transcript mentions "more than 128 megabytes", which is unclear). A small size delivers faster. The demo uses 1 (MB).
   • Buffer interval: how long before a not-yet-full buffer is flushed. 300 seconds waits five minutes. The demo uses 60 seconds, which the course called the minimum, so data is flushed at most every 60 seconds. The interval can be set to 0 (zero buffering, delivery within seconds). 900 seconds means about 15 minutes (the instructor says "I think").
6. Compression (GZIP, Snappy, Zip, Hadoop-compatible Snappy) saves space and cost. Encryption is optional.
7. Permissions: an IAM role is created automatically with the permissions to write to S3 and read from the Kinesis Data Stream.
8. Create it. Once active, the metrics, configuration and the error-log destination (CloudWatch Logs) are visible.
9. Test: Firehose only sees data sent after it is set up. In CloudShell, send three new records to DemoStream (user signup, user login, user logout) with the same put-record command from 09-Amazon-Kinesis-Data-Streams.
10. The S3 bucket has zero objects at first because of the 60 second buffer. After waiting, objects appear partitioned by date. One text file holds all three records.

Clean up

Delete the delivery stream first (type its name), then delete DemoStream. Otherwise it costs money every hour.

Questions & Answers