Streaming Pipeline Design AI Prompt

Design a streaming data pipeline for processing {{event_type}} events from {{source}} to {{destination}}.

Throughput requirement: {{throughput}} events/sec
Latency requirement: end-to-end < {{latency_target}}

1. Message broker configuration (Kafka / Kinesis):
   - Topic partitioning: number of partitions = max_throughput / throughput_per_partition
   - Partition key: choose a key that distributes load evenly AND ensures ordering where required
   - Retention: set to at least 7 days to allow replay from any point in the last week
   - Replication factor: 3 for production (tolerates 2 broker failures)

2. Consumer design:
   - Consumer group: one per logical pipeline to enable independent replay
   - Offset commit strategy: commit after successful write to destination (at-least-once delivery)
   - Idempotent consumer: handle duplicate messages at the destination with deduplication on event_id
   - Backpressure: limit consumer fetch size and processing batch to control memory usage

3. Stream processing (Flink / Spark Structured Streaming / Kafka Streams):
   - Windowing: tumbling window of {{window_size}} for aggregations
   - Watermark: allow late events up to {{late_arrival_tolerance}} before closing window
   - State management: use checkpointing every 60 seconds for fault tolerance

4. Exactly-once semantics:
   - Kafka transactions + idempotent producers for source-to-broker
   - Transactional writes to destination (or idempotent upserts)
   - Checkpoint-based recovery to avoid reprocessing

5. Dead letter queue:
   - Route unparseable or schema-invalid messages to a DLQ topic
   - Alert on DLQ growth rate > {{dlq_threshold}} messages/min

6. Monitoring:
   - Consumer lag per partition (alert if lag > {{lag_threshold}})
   - Processing latency (time from event timestamp to destination write)
   - Throughput (events/sec in and out)

Return: architecture diagram (text), configuration recommendations, processing code skeleton, and monitoring setup.