Workers

The Workers app (apps/workers) runs background processors that handle asynchronous tasks via BullMQ queues and PostgreSQL CDC streaming.

Starting Workers

# Development (with hot reload)
pnpm --filter @realtime/workers dev

# Production
pnpm --filter @realtime/workers start

Workers start automatically as part of pnpm dev via Turborepo.

Worker Processes

CDC Reader

Monitors your application database for row-level changes using PostgreSQL logical replication.

Flow:

PostgreSQL WAL → Logical Replication Slot → CDC Reader → Mapping Evaluator → Event Router

If CDC_DATABASE_URL is not configured, the CDC reader is disabled but all other workers run normally.

Outbox Worker

Processes the transactional outbox table (event_outbox) for guaranteed event delivery.

Flow:

event_outbox table → Outbox Worker → Event Router → Redis Pub/Sub → Clients

The outbox table uses a partial index on unprocessed rows for efficient polling. Processed rows are marked and cleaned up periodically.

Mapping Evaluator

Evaluates domain mappings against CDC events:

1. Receives a raw database change event
2. Looks up matching mappings by table name and operation type
3. Evaluates trigger column conditions (changed, eq, and, or)
4. Builds the event payload using $row.* template references
5. Forwards the normalized event to the Event Router

Webhook Delivery Worker

Delivers events to registered webhook endpoints via BullMQ:

• HMAC-SHA256 signing of every payload
• Exponential backoff with jitter on failure
• Max 5 retries per delivery
• Dead-letter queue for permanently failed deliveries
• Delivery logs recorded for every attempt

Analytics Worker

Buffers incoming events and periodically flushes aggregated analytics:

• Event counts by topic and source
• Latency percentiles
• Subscriber activity

Alert Worker

Evaluates alert rules against incoming events and metrics:

• Configurable alert rules with severity levels
• Cooldown periods to prevent alert storms
• Alert history for audit trails

Queue Architecture

All worker queues use BullMQ backed by Redis:

┌──────────────────────┐
│      Redis           │
│  ┌────────────────┐  │
│  │ webhook-delivery│  │
│  │ alerts          │  │
│  │ analytics-events│  │
│  │ report-generation│ │
│  └────────────────┘  │
└──────────┬───────────┘
           │
    ┌──────▼──────┐
    │   Workers   │
    │  Process    │
    │  jobs from  │
    │  queues     │
    └─────────────┘

Scaling Workers

Workers are stateless and horizontally scalable:

• Run multiple worker instances to increase throughput
• BullMQ handles job distribution and deduplication
• Each worker instance processes jobs from all queues
• No coordination required between instances

Monitoring Workers

• Check worker logs for startup messages and processing activity
• Use the Async Reliability dashboard in the Admin UI for queue metrics
• Monitor BullMQ queue depth, retry counts, and DLQ size
• Worker heartbeat status is visible in the Monitoring page

Troubleshooting