Data Flow & Architecture

This page provides a single unified view of every data path in the Realtime Platform — from event origination through delivery to clients and external systems — along with which components can be scaled independently.

Full System Diagram

┌─────────────────────────────────────────────────────────────────────────────────────────┐
│                                    DATA SOURCES                                         │
│                                                                                         │
│  ┌──────────────────┐    ┌──────────────────┐    ┌──────────────────┐                   │
│  │  Application DB   │    │  Client SDK /    │    │  Client SDK /    │                   │
│  │  (PostgreSQL)     │    │  REST API        │    │  REST API        │                   │
│  │                   │    │  sync.create()   │    │  socket.publish()│                   │
│  │  INSERT/UPDATE/   │    │  sync.merge()    │    │  broadcast()     │                   │
│  │  DELETE on tables │    │  sync.replace()  │    │                  │                   │
│  └────────┬─────────┘    └────────┬─────────┘    └────────┬─────────┘                   │
│           │                       │                       │                              │
│      WAL Stream              HTTP / WS                HTTP / WS                          │
└───────────┼───────────────────────┼───────────────────────┼──────────────────────────────┘
            │                       │                       │
            ▼                       ▼                       ▼
┌───────────────────┐  ┌────────────────────┐  ┌────────────────────────┐
│  CDC Pipeline     │  │  Sync Service      │  │  Socket Gateway        │
│  ┌──────────────┐ │  │  (HybridDocument   │  │  ┌──────────────────┐  │
│  │ WAL CDC      │ │  │   Service)         │  │  │ Socket.IO Server │  │
│  │ Reader       │ │  │                    │  │  │ (port 3000)      │  │
│  │              │ │  │  Redis hot cache +  │  │  └────────┬─────────┘  │
│  │ pgoutput     │ │  │  Postgres durable   │  │  ┌────────┴─────────┐  │
│  │ protocol     │ │  │  with revision      │  │  │ WebSocket Server │  │
│  └──────┬───────┘ │  │  tracking           │  │  │ (path: /ws)      │  │
│         │         │  └─────────┬──────────┘  │  └────────┬─────────┘  │
│         ▼         │            │              │           │            │
│  ┌──────────────┐ │            │              │           │            │
│  │ Mapping      │ │            │              │           │            │
│  │ Evaluator    │ │            │              │           │            │
│  │              │ │            │              │           │            │
│  │ • match table│ │            │              │           │            │
│  │ • check ops  │ │            │              │           │            │
│  │ • apply when │ │            │              │           │            │
│  │ • build      │ │            │              │           │            │
│  │   payload    │ │            │              │           │            │
│  └──────┬───────┘ │            │              │           │            │
│         │         │            │              │           │            │
└─────────┼─────────┘            │              └───────────┼────────────┘
          │                      │                          │
          │    RawEventInput     │    RawEventInput          │   Channel broadcast
          │    {topic,source,    │    {topic,source,          │   via ChannelService
          │     type,payload}    │     type,payload}          │
          ▼                      ▼                            ▼
┌──────────────────────────────────────────────────────────────────────────────────┐
│                              EVENT ROUTER                                        │
│                         (@realtime/event-router)                                 │
│                                                                                  │
│  ┌─────────────────┐   ┌─────────────────┐   ┌──────────────────────────────┐   │
│  │ EventNormalizer  │──▶│ EventPublisher   │──▶│ SubscriptionRegistry         │   │
│  │                  │   │                  │   │                              │   │
│  │ Assigns evt_ ID  │   │ Publishes to     │   │ In-memory Map<topic, Sub[]>  │   │
│  │ Wraps in         │   │ Redis Pub/Sub:   │   │                              │   │
│  │ RealtimeEvent    │   │ realtime:event:  │   │ • Looks up subs by topic     │   │
│  │ envelope         │   │ {topic}          │   │ • Applies EventFilter        │   │
│  │                  │   │                  │   │   (eq, in operators)          │   │
│  └─────────────────┘   └─────────────────┘   │ • Calls sub.handler()         │   │
│                                               │   for each match              │   │
│                                               └──────────────────────────────┘   │
└────────────────────────────────┬─────────────────────────────────────────────────┘
                                 │
                     Delivered to sub.handler()
                     (per-client callback)
                                 │
              ┌──────────────────┼──────────────────────┐
              │                  │                      │
              ▼                  ▼                      ▼
    ┌──────────────────┐  ┌────────────────┐  ┌──────────────────────┐
    │ Socket.IO Client │  │ Plain WS Client│  │  Side Effects        │
    │                  │  │                │  │                      │
    │ socket.emit(     │  │ ws.send(JSON)  │  │ See "Side Effect     │
    │   'event', data) │  │                │  │ Pipelines" below     │
    └──────────────────┘  └────────────────┘  └──────────────────────┘

    Additionally, the DeliveryDispatcher runs in parallel to deliver
    events to webhook endpoints, SQS queues, and SSE streams.
    See "Delivery Dispatcher" below.

Three Event Source Paths

Every event in the platform originates from exactly one of three sources. Each follows a distinct ingestion path before converging at the Event Router.

Path 1: Database CDC

Application DB (external PostgreSQL)
  │
  │  WAL logical replication (pgoutput protocol)
  │  Replication slot: realtime_cdc_slot
  │  Publication: realtime_cdc_pub
  │
  ▼
WalCdcReader (one per unique target DB URL)
  │
  │  Parses XLogData messages → extracts table, operation, old/new row
  │  Stores raw CdcEvent in cdc_events table (Postgres)
  │
  ▼
Mapping Evaluator (inside WalCdcReader.processChange)
  │
  │  For each active CdcSubscription on this table:
  │   1. Load DomainMapping from MappingService
  │   2. Check if operation matches mapping.events[]
  │   3. Evaluate mapping.when conditions (changed, eq, and/or)
  │   4. Build payload by resolving $row.column references
  │   5. Look up mapping.topics[action] → array of concrete topic names
  │   6. For each topic: Construct RawEventInput { topic, source:'database', type, payload }
  │
  ▼
EventRouter.route(input)
  │
  ▼
Clients receive event on subscribed topic

Key details:

CDC database URL is resolved per-application per-environment from Application.config.cdc[env]
Each unique target database gets its own WalCdcReader instance
Leader election (Redis SET NX) ensures only one backend instance holds the replication slot
Reader status is persisted in cdc_reader_status table (Postgres)

Path 2: Sync Service

Client SDK / REST API
  │
  │  POST /api/sync/{service}/documents         (create)
  │  PUT  /api/sync/{service}/documents/{id}     (replace)
  │  PATCH /api/sync/{service}/documents/{id}/merge  (merge)
  │  DELETE /api/sync/{service}/documents/{id}   (delete)
  │
  ▼
HybridDocumentService
  │
  │  1. Check Redis cache first (fast path)
  │  2. If cache miss → load from Postgres → cache in Redis with TTL
  │  3. Apply mutation (replace / deep-merge / delete)
  │  4. Compute diff between old and new data (JsonDiff)
  │  5. Increment revision number
  │  6. Check expectedRevision for optimistic concurrency (409 on conflict)
  │  7. Write to Redis immediately (refresh TTL)
  │  8. Async non-blocking flush to Postgres
  │
  ▼
publishEvent() → EventRouter.route()
  │
  │  topic: sync.{service}.{operation}
  │  source: 'sync'
  │  payload: { service, documentId, revision, data, diff }
  │  (diff included for update/delete; stripped per subscriber returnMode)
  │
  ▼
Clients receive event on subscribed topic

Key details:

Documents use a hybrid Redis + Postgres architecture: Redis as hot cache, Postgres as durable store
Active documents stay hot in Redis (default TTL: 10 minutes, configurable via SYNC_DOC_TTL_SECONDS)
Writes go to Redis immediately → async flush to Postgres (non-blocking)
Background dirty sweep (every 5s) ensures unflushed docs reach Postgres before TTL expiry
Falls back to Postgres-only when REDIS_URL is not set
Revision-based optimistic concurrency control
Topic naming convention: sync.{service}.created, sync.{service}.updated, sync.{service}.deleted

Path 3: Socket Service

Client SDK / REST API
  │
  │  socket.emit('subscribe', { topic })          (Socket.IO)
  │  ws.send({ type: 'subscribe', topic })        (plain WS)
  │  POST /api/socket/broadcast { channel, event, data }  (REST)
  │
  ▼
Socket Gateway (Socket.IO) / WS Gateway (plain WebSocket)
  │
  │  On subscribe:
  │   1. Generate sub_ prefixed ID
  │   2. Create Subscription { id, clientId, topic, filter, returnMode, handler }
  │   3. Register in SubscriptionRegistry (in-memory)
  │   4. Join Socket.IO room / track in WS channel map
  │   5. Handler wraps applyReturnMode() to strip data/diff per subscriber preference
  │
  │  On broadcast:
  │   1. ChannelService.broadcast(channel, event, data)
  │   2. Socket.IO: io.to(channel).emit(event, data)
  │   3. Plain WS: iterate wsSenders for channel members
  │   4. Fire broadcastHook → MetricsUpdater → HotTopicsAnalyzer
  │
  ▼
Directly delivered to channel members (no Event Router for raw broadcast)

Key details:

Two parallel gateways coexist on the same HTTP server: Socket.IO (default) and plain WebSocket (/ws path)
JWT authentication on connection handshake (both gateways)
When a client subscribes to a topic (not just a channel), the SubscriptionRegistry handler routes events from the Event Router directly to that client's socket

Outbox Pattern (Alternative to CDC)

Application code
  │
  │  BEGIN transaction
  │    INSERT INTO orders ...
  │    INSERT INTO event_outbox (topic, event_type, payload, table_name, operation)
  │  COMMIT
  │
  ▼
OutboxWorker (polling worker, runs in workers app)
  │
  │  SELECT * FROM event_outbox WHERE processed_at IS NULL
  │  ORDER BY id ASC LIMIT 100 FOR UPDATE SKIP LOCKED
  │
  │  For each row:
  │   1. Build RawEventInput from outbox row
  │   2. EventRouter.route(input)
  │   3. UPDATE event_outbox SET processed_at = NOW()
  │
  ▼
EventRouter.route(input) → Clients

Key details:

FOR UPDATE SKIP LOCKED enables multiple outbox workers to process concurrently without conflicts
Immediate re-poll when batch returns results (zero-delay tight loop), otherwise polls at configurable interval (default 1s)
Transactional guarantee: the outbox row and the business write are in the same transaction

Side Effect Pipelines

After events flow through the Event Router to clients, several side-effect systems process events asynchronously.

Delivery Dispatcher (Webhook, SQS, SSE)

The DeliveryDispatcher is the unified delivery pipeline for webhook, SQS, and SSE event delivery. It processes CDC events, system events (document CRUD, socket connect/disconnect), and any other events routed through the platform.

Event from any source (CDC, Sync, Socket, System)
  │
  ▼
DeliveryDispatcher.dispatch(event)
  │
  ├── Redis Subscription Index (O(1) entity key lookup)
  │   │  Falls back to Postgres scan if Redis unavailable
  │   ▼
  ├── Matching active delivery subscriptions
  │   │
  │   ├── Webhook → BullMQ queue (durable) or inline POST
  │   ├── SQS → BullMQ queue → SQS worker sends to AWS
  │   └── SSE → handled by SseConnectionManager (below)
  │
  └── SSE Broadcast (always, regardless of delivery subscriptions)
      │  SseConnectionManager checks all connected clients
      │  for token-based subscription matches
      ▼
      Matching SSE clients receive the event

Webhook Delivery

Gateway event (connect, disconnect, subscribe, broadcast)
  │
  ▼
WebhookDispatcher.dispatch(event)        ← fire-and-forget (non-blocking)
  │
  │  1. Query WebhookEndpointStore for active endpoints
  │     matching event type (or wildcard '*')
  │  2. Filter by applicationId + environment scope
  │  3. For each matching endpoint:
  │     a. JSON.stringify(event)
  │     b. HMAC-SHA256 signature: t={timestamp},v1={hash}
  │     c. POST to endpoint.url with retry (up to 3 attempts)
  │     d. Exponential backoff between retries
  │  4. Log result to delivery_logs table (Postgres)
  │  5. Fire onSuccess/onFailure → MetricsUpdater
  │
  ▼
External HTTP endpoint receives signed webhook payload

Webhook Delivery via BullMQ (Workers App)

Event enqueued to 'webhook-delivery' queue (BullMQ / Redis)
  │
  ▼
WebhookDeliveryWorker (in workers app)
  │
  │  1. Dequeue job from BullMQ
  │  2. Look up endpoint from WebhookRegistry
  │  3. Sign payload with HMAC-SHA256
  │  4. POST to endpoint URL with timeout + retry
  │  5. Exponential backoff: min(1000 * 2^attempt + jitter, 60s)
  │  6. Failed after maxAttempts → Dead Letter Queue (in-memory)
  │
  ▼
External HTTP endpoint

SQS Delivery via BullMQ

Event matched to SQS delivery subscription
  │
  ▼
DeliveryDispatcher.deliverSqs()
  │
  │  1. Validate queueUrl and region from subscription config
  │  2. Enqueue SQS delivery job to 'sqs-delivery' BullMQ queue
  │  3. SqsDeliveryWorker dequeues and sends to AWS SQS
  │  4. Supports both standard and FIFO queues
  │  5. FIFO: uses messageGroupId for ordering, auto-generates dedup ID
  │  6. Retry with exponential backoff (up to 3 attempts)
  │
  ▼
AWS SQS Queue receives message

Info

SQS delivery requires Redis (for BullMQ). Without a BullMQ queue, SQS deliveries are skipped with a warning log.

SSE Delivery

Event from any source
  │
  ▼
SseConnectionManager.broadcast(event)
  │
  │  1. Iterate all connected SSE clients on this instance
  │  2. Check applicationId + environment scope
  │  3. For each client's token subscriptions:
  │     a. Match event type against subscription patterns (exact, wildcard)
  │     b. Evaluate payload filters (AND logic on entity keys)
  │  4. Write SSE frame to matching client's HTTP response:
  │     id: evt_abc123
  │     event: proctoring.session.updated
  │     data: {"type":"proctoring.session.updated","subscriptionId":"ssub_001",...}
  │  5. Buffer event in Redis replay buffer (per token)
  │  6. Publish to Redis Pub/Sub for cross-instance delivery
  │
  ▼
SSE clients on ALL instances receive matching events

Cross-instance: When running multiple backend instances, each broadcasts to its local SSE clients. Redis Pub/Sub ensures events published on one instance reach SSE clients connected to other instances.

Replay buffer: Events are stored in a per-token Redis list (up to bufferSize entries). When a client reconnects with Last-Event-ID, missed events are replayed before live streaming resumes.

Analytics Pipeline

Events from any source
  │
  ▼
AnalyticsWorker (in workers app)
  │
  │  ingest(event) → buffer + rolling aggregations
  │
  │  Periodic flush (default 60s):
  │   • Drain buffer
  │   • Emit per-topic 1-minute aggregations
  │   • { topic, window:'1m', count, startTime, endTime }
  │
  ▼
Analytics storage / downstream consumers

Alert Pipeline

MetricsUpdater (5s tick in backend)
  │
  │  checkAlerts():
  │   • webhookFailureRate > 50% → critical alert
  │   • webhookFailureRate > 20% → warning alert
  │   • activeConnections > 10,000 → warning alert
  │
  ▼
AlertStore (in-memory, ring buffer of 1000)
  │
  ▼
GET /api/alerts → Admin UI Alert Center

Metrics Pipeline

Every gateway event, broadcast, webhook result
  │
  ▼
MetricsUpdater (5s tick)
  │
  │  Reads live state from:
  │   • ChannelService → connection count, channel list
  │   • SubscriptionRegistry → subscription count
  │   • CdcService → events received/routed, replication lag
  │   • WebhookDispatcher callbacks → success/failure counts
  │
  │  Computes rates and pushes into:
  │   • DashboardMetrics → delivery, reliability, pipeline metrics
  │   • HotTopicsAnalyzer → per-topic event count, fanout, latency
  │
  ▼
GET /api/metrics/dashboard → Admin UI Dashboard
GET /api/metrics/hot-topics → Admin UI Hot Topics
/metrics → Prometheus scrape endpoint

Redis Pub/Sub Cross-Instance Fanout

When running multiple backend instances, Redis Pub/Sub ensures events reach all connected clients regardless of which instance they're connected to:

Instance A                          Redis                         Instance B
┌──────────┐                   ┌────────────┐                   ┌──────────┐
│ CDC event │──EventPublisher──▶│ PUBLISH    │                   │          │
│ on topic  │                   │ realtime:  │──SUBSCRIBE────────▶│ Receives │
│ session.  │                   │ event:     │                   │ event,   │
│ status    │                   │ session.   │                   │ delivers │
│           │                   │ status     │                   │ to local │
│           │                   │            │                   │ subs     │
└──────────┘                   └────────────┘                   └──────────┘

Channel pattern: realtime:event:{topic} (e.g. realtime:event:session.status)

Data Stores

Store	Technology	Purpose	Scaled By
Platform DB	PostgreSQL	Metadata: topics, schemas, mappings, users, sessions, CDC status, delivery logs, audit logs, durable sync document storage, deployments	Vertical scaling, read replicas
Target DB(s)	PostgreSQL	Application databases monitored by CDC (external, one per app per env)	Independent of platform
Redis	Redis + RedisJSON	Hot sync document cache (TTL-based), Pub/Sub fanout, leader election locks, BullMQ job queues, presence	Redis Cluster for horizontal scaling

Independently Scalable Components

The platform is designed so each component can be scaled independently based on load characteristics:

┌─────────────────────────────────────────────────────────────────────┐
│                    SCALABILITY MAP                                   │
│                                                                     │
│  ┌─────────────────────────────────────────────┐                    │
│  │         Backend Instances (N)                │                    │
│  │                                              │                    │
│  │  ✅ REST API         — stateless, scale out  │                    │
│  │  ✅ Socket Gateway   — stateless with Redis  │                    │
│  │  ✅ WS Gateway       — stateless with Redis  │                    │
│  │  ⚠️  CDC Reader      — ONE leader instance   │                    │
│  │  ✅ Webhook Dispatch — async, non-blocking   │                    │
│  │  ✅ Metrics Updater  — per-instance local     │                    │
│  └─────────────────────────────────────────────┘                    │
│                                                                     │
│  ┌─────────────────────────────────────────────┐                    │
│  │         Worker Instances (M)                 │                    │
│  │                                              │                    │
│  │  ✅ Outbox Worker    — scale with SKIP LOCKED│                    │
│  │  ✅ Webhook Worker   — scale via BullMQ      │                    │
│  │  ✅ Analytics Worker — independent flush      │                    │
│  │  ✅ Alert Worker     — per-instance rules     │                    │
│  └─────────────────────────────────────────────┘                    │
│                                                                     │
│  ┌─────────────────────────────────────────────┐                    │
│  │         Infrastructure                       │                    │
│  │                                              │                    │
│  │  ✅ PostgreSQL       — read replicas          │                    │
│  │  ✅ Redis            — Redis Cluster          │                    │
│  │  ✅ Admin UI         — static SPA, CDN        │                    │
│  └─────────────────────────────────────────────┘                    │
└─────────────────────────────────────────────────────────────────────┘

Component Scaling Details

Scale: Horizontally (add more instances behind a load balancer)

The backend is stateless — all persistent state lives in PostgreSQL and Redis. Adding more instances increases:

HTTP request throughput (REST API)
WebSocket connection capacity (Socket.IO + plain WS)
Webhook dispatch parallelism

Constraint: CDC readers use Redis-based leader election. Only one instance holds the replication slot at a time. If the leader dies, another instance acquires the lock within ~5 seconds (heartbeat 3s, follower poll 5s, plus Redis pub/sub signal for immediate failover).

Load Balancer (sticky sessions for WS)
     │
┌────┼────┬────────┐
▼    ▼    ▼        ▼
B1   B2   B3  ...  Bn
│    │    │        │
└────┴────┴────────┘
          │
     Redis Pub/Sub
(cross-instance fanout)

Scale: Horizontally (add more worker processes)

Workers are fully independent and share no in-process state:

Worker	Scaling Mechanism	Concurrency Safety
Outbox Worker	`FOR UPDATE SKIP LOCKED` — multiple workers poll the same table without conflicts	Row-level locking
Webhook Worker	BullMQ job distribution — Redis-backed queue with automatic job assignment	Queue-level dedup
Analytics Worker	Independent buffer + flush cycle per instance	Eventually consistent
Alert Worker	Per-instance rule evaluation, shared alert store	Cooldown per rule

Scale: Vertically (one leader, failover to standby)

PostgreSQL logical replication slots are single-consumer — only one reader can consume from a slot at a time. The platform handles this via Redis-based leader election:

Redis Key: realtime:cdc:leader
┌────────────────────────────────────────┐
│  SET NX + PX (10s TTL)                 │
│  Leader heartbeat: every 3s            │
│  Follower poll: every 5s               │
│  Pub/Sub signal: realtime:cdc:leader:  │
│  signal (immediate failover notify)    │
└────────────────────────────────────────┘

Instance A (LEADER)  ──reads WAL──▶  Target DB
Instance B (FOLLOWER) ──waiting──
Instance C (FOLLOWER) ──waiting──

If A dies → B or C acquires lock within ~5s

Each unique target database URL gets its own WalCdcReader. Multiple applications can point to different databases, and each gets an independent reader — but each reader is still single-consumer per replication slot.

Event Envelope Reference

Every event flowing through the platform uses this normalized envelope:

interface RealtimeEvent {
  id: string;                          // e.g. evt_a1b2c3d4
  topic: string;                       // e.g. session.status
  source: 'database' | 'sync' | 'socket';
  type: string;                        // e.g. database.update
  payload: Record<string, unknown>;   // sync events: { service, documentId, revision, data?, diff? }
  metadata: {
    timestamp: number;
    table?: string;                    // database events only
    operation?: 'insert' | 'update' | 'delete';
    revision?: number;                 // sync events only
  };
}

End-to-End Latency Breakdown

Database CDC Path (target: < 150ms end-to-end):
  WAL write → pgoutput decode    ~10-30ms
  Mapping evaluation             ~1-5ms
  EventRouter normalize+publish  ~5-15ms
  Redis Pub/Sub fanout           ~1-5ms
  Socket delivery to client      ~5-20ms

Sync Service Path (target: < 50ms):
  HTTP request parsing           ~1-2ms
  Redis cache read (hit)         ~1-3ms   (or Postgres fallback ~10-25ms)
  Redis cache write + TTL        ~1-3ms
  Async Postgres flush           ~0ms     (non-blocking, background)
  EventRouter normalize+publish  ~5-15ms
  Socket delivery to client      ~5-20ms

Socket Broadcast Path (target: < 100ms):
  Message received by gateway    ~1-2ms
  Channel broadcast via IO       ~5-30ms (depends on subscriber count)
  Redis cross-instance fanout    ~1-5ms (if multi-instance)

Complete Wiring Summary

This table shows how every major component is wired together at startup in apps/backend/src/index.ts:

Component	Created From	Wired To
`EventRouter`	noopPublisher + subscriptionRegistry	CdcService, HybridDocumentService
`SubscriptionRegistry`	standalone (in-memory)	EventRouter, Socket Gateway, WS Gateway, Socket API
`ChannelService`	Socket.IO Server	Socket Gateway, WS Gateway, MetricsUpdater
`CdcService`	stores + applicationService + mappingService + eventRouter	CDC API, MetricsUpdater
`WalCdcReader`	created on-demand by readerFactory	CdcService (per target DB)
`CdcLeaderElection`	Redis clients (main + sub)	CdcService (optional)
`WebhookDispatcher`	webhookStore + logStore + callbacks	Socket Gateway, WS Gateway
`DeliveryDispatcher`	deliverySubscriptionStore + webhookQueue + sqsQueue + sseManager	CDC events, Sync events, System events
`SseConnectionManager`	Redis client	SSE API routes, DeliveryDispatcher
`RedisSubscriptionIndex`	Redis client	DeliveryDispatcher (O(1) entity key lookups)
`MetricsUpdater`	dashboard + hotTopics + channelService + subscriptionRegistry	Broadcast hook, periodic 5s tick
`HybridDocumentService`	RedisJsonClient + Knex db + EventRouter (when REDIS_URL set)	Sync API routes
`PgDocumentService`	Knex db (fallback when no REDIS_URL)	Sync API routes
`JwtService`	SigningKeyStore (pre-loaded from Postgres)	WS Gateway auth, Auth middleware