Event-Driven Integration with Symfony Messenger and Redis Streams

The client was a global manufacturing brand managing over 50 web properties across 14 country markets. Product catalogue, pricing, and inventory data originated in SAP S/4HANA and needed to propagate to every market website — historically via synchronous RFC calls from a Symfony middleware layer that had grown brittle over 8 years.

The symptoms were textbook: cascading failures when SAP was slow, inconsistent data states between markets, deployment windows limited to nights and weekends, and a middleware team of six spending 70% of their time on incident response. The solution was a full architectural shift to event-driven integration.

Why Symfony Messenger Over Kafka

This is the question we get asked most. For this client's scale (approximately 15,000 events/hour peak), Kafka would have been architectural over-engineering. Kafka's operational complexity — ZooKeeper/KRaft management, partition design, consumer group rebalancing — would have required dedicated infrastructure expertise the team didn't have.

Redis Streams with Symfony Messenger provided:

• Persistent, ordered event log with consumer group semantics (XREADGROUP)
• Native Symfony integration without custom transport code
• Operational simplicity — Redis is already in the stack for caching
• Sufficient throughput for the use case with horizontal worker scaling

The decision rule we apply: if you need Kafka's throughput (millions of events/second), use Kafka. If you need Kafka's semantics at lower volume, Redis Streams is operationally simpler and sufficient.

The Dead-Letter Queue Pattern

The most critical reliability feature in the implementation is the dead-letter queue (DLQ) strategy. Event-driven architectures fail silently when messages are lost. We implemented a three-tier retry and DLQ pattern:

Failed messages move to a retry stream with exponential backoff (30s, 5min, 30min). After three retries, they move to the DLQ. A separate monitoring process alerts the team via PagerDuty when DLQ depth exceeds threshold. Manual review can replay DLQ messages after the root cause is resolved.

SAP Change Data Capture

The upstream feed from SAP was implemented using SAP's Change Document framework rather than polling. An ABAP function module publishes change events to an HTTP endpoint on state changes to relevant business objects (materials, prices, inventory). This push model eliminated the polling overhead and reduced event latency from ~15 minutes (scheduled polling) to under 30 seconds.

A dedicated Symfony "ingestor" service validates, normalises, and enriches incoming SAP events before publishing them to Redis Streams. This separation of concerns means the event schema in the streams is decoupled from SAP's native data structures — an important boundary for maintainability.