Event-Driven Architecture Development

Apache Kafka cluster design, topic architecture, and producer/consumer implementation for high-throughput event streaming at production scale. Partition strategy designed around your ordering requirements: the partition key is the entity ID (order ID, customer ID, device ID) so all events for the same entity land in the same partition and are processed in order by the consumer, while events for different entities are processed in parallel across partitions. Consumer group topology: each distinct processing concern (fulfilment, analytics, notifications) gets its own consumer group reading the same topic independently, with separate committed offsets so a slow analytics consumer doesn't block the real-time fulfilment consumer. Confluent Schema Registry (or AWS Glue Schema Registry) for schema management with Avro or Protobuf encoding: backward-compatible schema evolution lets producers add fields without breaking existing consumers, and forward-compatible evolution lets consumers handle events from future producers. Topic configuration: replication factor 3 with min.insync.replicas=2 for durability; retention period set to match your replay window (7 days for operational replay, 90 days for audit trails); log compaction for event-sourced entity topics that only need the latest event per key. Producer idempotency and transactional APIs for exactly-once semantics when the same event must not be produced twice even if the producer restarts. Managed Kafka on Confluent Cloud, AWS MSK, or Aiven, or self-managed on Kubernetes via Strimzi operator depending on your control requirements and operational budget.

RabbitMQ exchange, queue, and binding topology designed for your routing complexity and reliability requirements. Exchange type selection by use case: direct exchange for point-to-point task distribution (payment processing, email sending), fanout exchange for broadcasting the same message to multiple queues (new-order event that triggers fulfilment, inventory, and notification queues simultaneously), topic exchange for pattern-based routing (order.placed.US.enterprise routed to separate queues for regional and customer-tier processing). Dead letter queue configuration: messages that are rejected, expire, or exceed the maximum delivery attempt count are routed to a dead letter exchange and queue for inspection and replay without being silently discarded, the observability that turns "that message never arrived" into a visible, recoverable failure. Message TTL policies per queue: time-sensitive messages (notifications, live data updates) expired after a configurable window so stale messages don't process after they lose meaning. Consumer prefetch count (QoS setting) calibrated to the consumer's processing speed to prevent a fast producer overwhelming a slow consumer with unbounded in-memory messages. Consumer acknowledgement modes: manual ack with requeue-on-failure for critical processing (the message stays in the queue until the consumer explicitly confirms it was processed), auto-ack for fire-and-forget notifications where duplicate delivery is acceptable. RabbitMQ Cluster with quorum queues for HA: quorum queues replicate messages across most cluster nodes, surviving node loss without message loss. CloudAMQP or RabbitMQ on Kubernetes (via the RabbitMQ Cluster Operator) depending on operational preference.

Event sourcing architecture where each state change is stored as an immutable event in an append-only log rather than updating a mutable row, giving you a complete, replayable audit trail of everything that has ever happened to every entity in the system, with the ability to reconstruct any historical state by replaying events to a point in time. Event store implementation using EventStoreDB (purpose-built event store with built-in projections and subscriptions) or PostgreSQL with an events table (aggregate_id, event_type, sequence_number, payload, created_at) depending on scale and operational preference. Snapshot strategy for aggregates with long event histories: every N events, a snapshot of the current state is written so replaying a high-event-count aggregate starts from the nearest snapshot rather than replaying from the beginning each time. CQRS (Command Query Responsibility Segregation) implemented alongside event sourcing: the write side (command handlers) processes commands, raises events, and appends them to the event store; the read side (projections) subscribes to the event stream and maintains denormalised read models optimised for each query pattern, an order status view, a customer history view, and an analytics aggregate each maintained independently without the join overhead that would be required from a normalised write model. Projection rebuild capability: when a new read model is needed or an existing projection requires a schema change, the projection is rebuilt by replaying the event stream from the start, the ability to ask new questions of historical data without retroactive migration.

Event bus topology for systems where multiple services need to react to the same business events without knowing about each other, the architecture that lets you add a new service that reacts to order.placed without modifying the order service or any other existing subscriber. Domain event catalog: every business event named using past-tense domain language (OrderPlaced, PaymentCaptured, ShipmentDispatched), with the event type, version, schema, producing service, and consuming services documented in a contract registry. Event envelope structure: a standard wrapper (event_id, event_type, aggregate_id, aggregate_type, occurred_at, schema_version, payload) applied to every event regardless of domain, making serialisation, routing, and consumer filtering consistent across the system. Schema evolution policy: v1 events remain supported until all consumers have migrated, new required fields added as optional with a defined deadline for consumer updates, breaking changes published as v2 events with a migration period where both versions are produced. CloudEvents standard specification used as the event envelope format where cross-organisation or cross-platform interoperability is required. Consumer contract testing with Pact: consumer publishes its expectations of the event schema, producer CI validates that the produced event satisfies all registered consumer contracts before merging, preventing the producer from shipping a schema change that silently breaks a consumer in a separate repository.

Long-running business process orchestration for workflows that span multiple services, take minutes to hours to complete, and must recover gracefully when any step fails. Saga pattern selection by use case: choreography saga (each service publishes events and reacts to others without a central coordinator, appropriate for simple linear workflows) or orchestration saga (a central orchestrator sends commands to services and waits for completion events, appropriate for complex branching workflows where visibility matters). Compensating transaction design: each step in the saga has a defined compensation action that undoes the step's effect if a later step fails; an order saga that fails at the payment step cancels the inventory reservation it already applied, maintaining system consistency without a distributed transaction. Temporal workflow engine for complex long-running orchestration: Temporal provides durable execution (workflows survive application restarts and infrastructure failures), deterministic replay (the workflow's history can be replayed to diagnose failures), and built-in retry with configurable backoff per activity. Conductor (Netflix) as an alternative for organisations already invested in the Java ecosystem. State machine implementation for approval workflows and processing pipelines where each state is explicit, transitions are defined, and the current state of every in-flight process is queryable. Timeout and escalation handling: a step that doesn't complete within a configurable SLA triggers an escalation event (alert, reassignment, fallback action) rather than silently hanging.

Event-driven integration layers connecting external APIs, SaaS platforms, and legacy systems to your internal event infrastructure, bridging the gap between systems that push webhooks, systems that require polling, and systems that expose database-level change streams. Webhook ingestion service: receives incoming webhooks from Stripe, GitHub, Shopify, or any HTTP-based external source; validates the HMAC signature; publishes to an internal Kafka topic; responds HTTP 200 immediately, decoupling the external caller from internal processing latency, handling burst webhook delivery (Black Friday Stripe webhooks) by queuing rather than dropping, and providing automatic retry on downstream consumer failure. Change data capture (CDC) with Debezium: connects to your PostgreSQL, MySQL, or MongoDB database and streams every row-level INSERT, UPDATE, and DELETE to a Kafka topic in real time, enabling downstream services to react to database changes without polling queries or application-level event publishing hooks. Debezium runs as a Kafka Connect connector, managed on Confluent Cloud or self-hosted. Transactional outbox pattern implementation for systems where application-level event publishing must be consistent with the database write: the application writes the event to an outbox table in the same database transaction as the domain state change; a separate relay process reads the outbox table and publishes to Kafka; the published event is acknowledged and deleted from the outbox, eliminating the dual-write problem where a successful database write followed by a failed event publish produces inconsistent state.