Parallelizing Polling Across Entities with RabbitMQ

Problem

One polling message triggered sequential processing of all entities. A slow provider (5s) blocked all fast providers (0.1s). Pods 2 and 3 sat idle.

Fix: publish one message per entity

flowchart TD
    subgraph before [Before: 1 message]
        M1[1 message] --> P1[Pod 1]
        P1 --> A1[Provider A 5s]
        A1 --> B1[Provider B 0.1s]
        B1 --> C1[Provider C 0.1s]
        C1 --> D1[Provider D 0.1s]
    end

    subgraph after [After: N messages]
        MA[msg A] --> PA["Pod 1<br/>Provider A 5s"]
        MB[msg B] --> PB["Pod 2<br/>Provider B 0.1s"]
        MC[msg C] --> PC["Pod 3<br/>Provider C 0.1s"]
        MD[msg D] --> PB
    end

Let RabbitMQ’s competing consumers do the load balancing.

Implementation changes

1. Redis key — include entity ID so each entity gets its own idempotency key
2. Publisher — loop through enabled entities, publish one message per entity
3. Consumer — process the single entity from the message, not all entities from DB

Why not separate queues per provider?

• 10 providers = 10 queues to monitor, configure, scale separately
• Breaks RabbitMQ’s natural competing-consumer load balancing
• Adding a new provider becomes an infrastructure change instead of a data change

See also

• setnx-dedup-multi-pod — prevents duplicate messages from multiple pods
• panic-recovery-dlq — handling panics in the consumer that processes these messages