Amazon Kinesis – Overview
Amazon Kinesis is a real-time data streaming service designed for ingesting, processing, and analyzing large volumes of streaming data (e.g., logs, clickstreams, IoT).
Kinesis Data Streams (KDS)
A low-latency, scalable stream for real-time processing of data records.
- Data is written to shards
- Consumers read using Kinesis Client Library (KCL) or AWS Lambda
- Default retention: 24 hours, extendable to 7 days
Kinesis Data Streams – Capacity Modes
| Mode | Description |
|---|---|
| Provisioned | You set the number of shards. Each shard supports 1 MB/s write, 2 MB/s read. |
| On-Demand | Auto-scales based on traffic. Best for unpredictable or spiky workloads. |
Shards determine throughput capacity.
Kinesis Data Streams – Security
| Feature | Description |
|---|---|
| IAM Policies | Control access to streams (read/write) |
| KMS Encryption | Server-side encryption at rest |
| VPC Endpoints (PrivateLink) | Secure communication within your network |
| HTTPS Ingestion | Protects data in transit |
Kinesis Data Firehose
A fully managed delivery service to load streaming data into destinations like:
- Amazon S3
- Redshift
- OpenSearch
- Splunk
- Custom HTTP endpoints
No need to manage shards. Firehose buffers, transforms, batches, and delivers data automatically.
Kinesis Data Streams vs Firehose
| Feature | Kinesis Data Streams | Kinesis Firehose |
|---|---|---|
| Use Case | Real-time processing | Near real-time delivery to storage |
| Management | Manual shard setup | Fully managed, auto-scaled |
| Latency | <1 second | 60–90 seconds (buffering) |
| Data Consumers | Apps, Lambda, KCL | S3, Redshift, OpenSearch, etc. |
Ordering Data into Kinesis
- Ordering is guaranteed per shard
- Use a consistent partition key to ensure related events go to the same shard
Ordering Data into SQS
- Standard SQS: Best-effort ordering, not guaranteed
-
FIFO SQS: Guaranteed ordering with
MessageGroupId
Kinesis vs SQS – Ordering
| Feature | Kinesis | SQS |
|---|---|---|
| Ordering | Per shard | FIFO queues only |
| Latency | Sub-second | Tens of milliseconds to seconds |
| Consumers | Multiple (fan-out) | 1:1 or fan-out via SNS |
SQS vs SNS vs Kinesis
| Feature | SQS | SNS | Kinesis |
|---|---|---|---|
| Type | Message Queue | Pub/Sub Notification | Streaming Platform |
| Push vs Pull | Pull | Push | Pull (via consumers) |
| Message Retention | Up to 14 days | No retention | 24h–7 days |
| Ordering | FIFO (optional) | Not guaranteed | Per shard |
| Use Case | Decoupling workers | Notifications, Fan-out | Real-time data processing |
Amazon MQ (Managed Message Broker)
- Fully managed Apache ActiveMQ or RabbitMQ
- For legacy systems or apps needing:
- JMS, AMQP, STOMP, MQTT
- Supports durable queues, topics, transactional messages
- Not serverless — runs on EC2 infrastructure
Use Amazon MQ if you're migrating from on-premise enterprise message brokers.
When to Use What?
| Need | Use |
|---|---|
| Real-time analytics | Kinesis Data Streams |
| Archival/logging pipelines | Kinesis Firehose |
| Simple decoupled messaging | SQS (Standard or FIFO) |
| Instant push notifications | SNS |
| Message broker compatibility | Amazon MQ |