Real-Time Data Pipeline Development | Kafka

Streaming platform selection and setup based on your cloud environment, throughput requirements, and operational capacity: Apache Kafka on AWS MSK (managed Kafka, removes broker management overhead, supports multi-AZ replication, scales partition count without downtime) for high-throughput event streaming; AWS Kinesis Data Streams for AWS-native teams who prefer fully managed infrastructure with a pay-per-shard pricing model; Google Cloud Pub/Sub for GCP-native teams or those needing global fan-out across regions; or self-managed Kafka on Kubernetes for organisations that need full control over broker configuration and storage. Topic design: one topic per event type with a naming convention (domain.entity.event -- e.g., payments.transaction.created, inventory.sku.updated), partition count set based on target throughput (Kafka benchmark: ~10--50 MB/s per partition on standard MSK instances) and required consumer parallelism (partition count = max consumer instances). Partition key selection to preserve event ordering for entities that require it (all transaction events for a given customer ID go to the same partition to guarantee order). Replication factor 3 across availability zones for durability; retention policy set to the replay window requirement (7--30 days for most operational pipelines; indefinite for event sourcing architectures). Producer configuration: acks=all for financial data requiring durability guarantees; linger.ms and batch.size tuned for throughput vs latency trade-off based on your use case. Infrastructure-as-code via Terraform or AWS CDK for reproducible cluster setup and configuration management.

Apache Flink for stateful stream processing where the transformation logic requires aggregating across multiple events, handling out-of-order arrivals, or maintaining running computations over a time window. Flink's event-time processing model handles late-arriving events correctly -- an event with a timestamp from 30 seconds ago that arrives late due to network delay is placed in the correct time window rather than dropped or incorrectly attributed to the current window, which matters for session analytics and fraud scoring. Processing patterns implemented: event filtering (discard low-priority events before they consume downstream storage); data enrichment (join each event with reference data from a Redis cache or PostgreSQL lookup to add customer tier, product category, or geographic region before forwarding); tumbling window aggregation (count, sum, or average over fixed non-overlapping 1-minute or 5-minute windows, used for metric computation); sliding window aggregation (rolling 15-minute or 1-hour windows with 1-minute slide, used for moving average calculations in fraud and anomaly detection); and session windows (group user events into sessions with a configurable inactivity timeout, used for user behaviour analysis). Complex event processing (CEP) with Flink's pattern API: detect sequences of events that indicate a fraud pattern (three failed logins followed by a successful login followed by a high-value transaction within 5 minutes) and emit a fraud alert event without storing the raw events in the alert system. For simpler transformations without stateful logic, AWS Lambda functions or Kafka Streams are more operationally lightweight alternatives. Processing job deployment: on AWS EMR (Spark Streaming), AWS Kinesis Data Analytics (Flink-as-a-service), or self-managed Flink cluster on Kubernetes.

Change Data Capture with Debezium reads directly from the database transaction log rather than polling application tables -- capturing every insert, update, and delete at the row level with sub-millisecond latency from database commit to Kafka topic, without adding query load to the production database. Database support: PostgreSQL (logical replication with pgoutput or decoderbufs plugin, requires wal_level=logical); MySQL and MariaDB (binlog with row-based format, requires binlog_format=ROW); Microsoft SQL Server (SQL Server Agent CDC or Always On availability groups); Oracle (LogMiner); and MongoDB (change streams). Each row-level change is published to Kafka as a structured event with the before-state and after-state of the row, the operation type (c/u/d/r for create/update/delete/read), the transaction ID, and the source database position. Exactly-once semantics from database to Kafka using Debezium's transaction metadata: events within a single database transaction are published atomically. Schema change handling: Debezium integrates with Confluent Schema Registry (Avro or JSON Schema) to manage schema evolution; when a column is added or renamed in the source database, the schema registry version is updated and consumers receive the schema change alongside the data events. CDC use cases: real-time data replication from production OLTP database to analytical data warehouse (no ETL batch window); audit log generation capturing every data change without application-level logging; event-driven microservices (the outbox pattern where application writes to a Kafka topic row in the same transaction as the business object, and CDC publishes the event transactionally).

Real-time feature pipelines for ML models that require current feature values at inference time -- the category where stale batch features produce demonstrably worse model performance. Fraud detection models need features computed from the last 60 seconds of transaction activity (transaction count in the last 5 minutes, total amount in the last hour, number of distinct merchants in the last 24 hours) -- features that a nightly batch pipeline cannot provide. Feature store platforms: Feast (open source, Kubernetes-deployable, supports Redis online store and BigQuery/Snowflake offline store) for teams that want full control and no vendor lock-in; Tecton (managed service, handles streaming feature computation natively with Flink under the hood) for teams that prefer a managed platform; or a custom Redis-backed feature store for simple feature sets where a full feature platform is unnecessary overhead. Streaming feature computation pipeline: Flink jobs read from Kafka event topics, compute time-windowed aggregations (count_last_1h, sum_last_24h, distinct_count_last_7d), and write computed features to the online store (Redis, DynamoDB) with sub-second latency. Feature freshness guarantees: each feature has a configured maximum staleness SLA; the monitoring layer alerts when feature store entries exceed their staleness threshold (the online store has not been updated recently enough). Batch-streaming consistency: the same feature logic is implemented in both the streaming Flink job and the batch dbt model that backfills historical features for model training -- ensuring the features the model was trained on are the same features it sees at inference. Point-in-time correct feature retrieval for model training data generation via Feast's time-travel query.

Delivery from the streaming layer to multiple operational destinations -- each consumer group tracks its own Kafka offset independently, so a slow or failing consumer does not block other consumers or create backpressure against the upstream producer. Destination patterns by use case: PostgreSQL or Aurora for application reads where the streaming data needs to be queryable via standard SQL (upsert-based writes with conflict resolution on the primary key); Amazon DynamoDB or MongoDB for high-throughput, low-latency key-value lookups (fraud decisions, user preference reads); Elasticsearch or OpenSearch for full-text search indexing and real-time alerting (streaming events indexed as Elasticsearch documents with configurable mapping, powering search APIs and Kibana alert rules); Redis for sub-millisecond application cache updates (session data, rate limit counters, real-time leaderboards); Snowflake, BigQuery, or Redshift via Kafka connectors (Confluent Kafka Connect Snowflake Sink, BigQuery Storage Write API) for real-time analytical warehouse loading with configurable micro-batch intervals (30 seconds to 5 minutes); and Apache Pinot or Druid for OLAP queries over streaming data where sub-second query latency on recent events is required. Fan-out architecture: a single event stream on a Kafka topic is consumed by multiple independent consumer groups -- the fraud service, the analytics pipeline, the audit logger, and the operational dashboard all read the same stream without duplication at the source or coordination between consumers. Kafka consumer group lag per destination is monitored separately, with independent alerting thresholds appropriate for each destination's SLA.

Pipeline monitoring covers the metrics that predict failures before they breach SLAs, not just the metrics that confirm they already have. Consumer lag monitoring per consumer group per partition published to Prometheus and visualised in Grafana: lag threshold alerts are set based on the production rate (at 50,000 events/minute with a 5-minute lag tolerance, alert at 250,000 events lag) rather than arbitrary fixed numbers. Alerting via PagerDuty for pipeline-down incidents, Slack for non-critical degradation. Dead letter queue (DLQ): events that fail processing after the configured retry count (typically 3 with exponential backoff) are written to a dedicated DLQ topic (e.g., payments.transaction.created.dlq) with the original event, the failure reason, the retry count, and the timestamp. The DLQ is monitored separately -- a non-empty DLQ is always a page to the on-call engineer. DLQ events are reprocessable: after a bug fix, events are replayed from the DLQ through the corrected processor. Backpressure handling: when a downstream system (e.g., a PostgreSQL destination) slows down under write pressure, the consumer pauses partition polling to prevent unbounded memory growth in the consumer process; lag accumulates on the topic (handled by Kafka's durable retention) rather than in memory. End-to-end latency measurement: synthetic probe events injected at configurable intervals measure the full pipeline latency from producer publish to downstream destination write -- alerting when end-to-end latency exceeds the SLA (e.g., 30 seconds for an operational dashboard, 5 seconds for a fraud detection pipeline). Operations runbook covering consumer lag investigation, consumer group reset, DLQ reprocessing procedure, and broker failure recovery -- delivered with the project.