Architecture

The Realtime Platform is a modular, horizontally-scalable system designed around domain-driven subscriptions and a normalized event protocol.

Design Principles

Stateless Infrastructure

Application nodes are stateless. All persistent state lives in:

PostgreSQL — metadata, migrations, audit trails, durable sync document storage (sync_documents table)
Redis — hot document cache (RedisJSON with TTL), pub/sub fanout, presence, queues (BullMQ)

This enables horizontal scaling by simply adding more API server or worker instances.

Hybrid Document Storage

Sync documents use a write-through cache pattern for optimal latency and durability:

Reads: Redis first (fast path) → cache miss falls back to Postgres → loads result into Redis with TTL
Writes: Written to Redis immediately → asynchronously flushed to Postgres (non-blocking)
TTL: Active documents stay hot in Redis (default: 10 minutes, configurable via SYNC_DOC_TTL_SECONDS). Every read/write refreshes the TTL.
Dirty sweep: A background sweep (every 5s) persists unflushed documents to Postgres before TTL expiry.
Shutdown: Final flush on graceful shutdown ensures no data loss.
Fallback: When REDIS_URL is not set, the system uses Postgres-only storage automatically.

Clients subscribe to domain topics (e.g. session.status, order.updated), never to internal service details. The platform determines the producing service (Sync, Socket, or Database CDC) internally and delivers events through a unified envelope.

Explicit Mutations

Writes target the responsible service directly:

realtime.sync.mutate(...)    // Sync service
realtime.socket.publish(...) // Socket service

Database writes never occur through the platform — the platform only observes database changes via CDC.

System Architecture

┌──────────────────────────────────────────────────────────────┐
│                        Clients (SDK)                         │
│  subscribe(topic) · sync.create() · socket.publish()         │
└──────────────┬──────────────────────────────┬────────────────┘
               │ HTTP/WS                      │ HTTP/WS
┌──────────────▼──────────────────────────────▼────────────────┐
│                   Backend (Express + Socket.IO)               │
│                                                               │
│  ┌─────────┐  ┌──────────┐  ┌──────────┐  ┌──────────────┐  │
│  │ REST API│  │ WS Gateway│  │ Services │  │  Middleware   │  │
│  │ Routes  │  │ Events   │  │ Sync/DB/ │  │ Auth/Scope   │  │
│  │         │  │          │  │ Socket   │  │              │  │
│  └────┬────┘  └────┬─────┘  └────┬─────┘  └──────────────┘  │
│       └─────────────┼────────────┘                            │
│                     │                                         │
│              ┌──────▼──────┐                                  │
│              │ Event Router │                                  │
│              │ Normalize → │                                  │
│              │ Publish →   │                                  │
│              │ Filter →    │                                  │
│              │ Deliver     │                                  │
│              └──────┬──────┘                                  │
└─────────────────────┼────────────────────────────────────────┘
                      │
         ┌────────────┼────────────┐
         │            │            │
    ┌────▼───┐  ┌─────▼────┐  ┌───▼────┐
    │ Redis  │  │PostgreSQL│  │ BullMQ │
    │JSON/PS │  │ Metadata │  │ Queues │
    └────────┘  └──────────┘  └───┬────┘
                                  │
                      ┌───────────▼──────────┐
                      │     Workers          │
                      │ CDC · Outbox ·       │
                      │ Webhooks · Analytics │
                      └──────────────────────┘

Event Flow

Every event in the system follows the same lifecycle:

Origination

An event originates from one of three sources:

Database CDC — a row change in a monitored table
Sync Service — a document create/update/delete
Socket Service — a published message

Normalization

The EventNormalizer converts the raw event into a standard RealtimeEvent envelope with a unique ID, topic, source, and metadata.

Publishing

The EventPublisher sends the normalized event to the Redis Pub/Sub channel for the topic: realtime:event:{topic}

Routing

The EventRouter receives published events and:

Looks up active subscriptions for the topic
Applies subscription filters (equality, in operator)
Delivers matching events to connected clients via Socket.IO

Side Effects

Webhook endpoints, analytics workers, and alert workers process events asynchronously via BullMQ queues.

Event Envelope

All services communicate using a normalized envelope:

interface RealtimeEvent {
  id: string;                          // e.g. evt_a1b2c3d4
  topic: string;                       // e.g. session.status
  source: 'database' | 'sync' | 'socket';
  type: string;                        // e.g. database.update
  payload: Record<string, unknown>;
  metadata: {
    timestamp: number;
    table?: string;                    // database events
    operation?: 'insert' | 'update' | 'delete';
    revision?: number;                 // sync events
  };
}

Redis channel pattern: realtime:event:{topic} (e.g. realtime:event:session.status)

Monorepo Organization

The codebase is organized as a pnpm monorepo with Turborepo orchestration:

realtime/
├── apps/                    # Deployable applications
│   ├── backend/             # Express API + Socket.IO
│   ├── workers/             # Background processors
│   ├── admin-ui/            # React management UI
│   └── docs/                # Documentation (this site)
├── packages/                # Shared libraries
│   ├── shared-types/        # TypeScript interfaces
│   ├── shared-utils/        # Utilities (errors, IDs, etc.)
│   ├── shared-config/       # Zod-validated env config
│   ├── observability/       # Logging (pino) + Prometheus
│   ├── redis-layer/         # Redis client + pub/sub + JSON
│   ├── auth/                # JWT + signing keys + middleware
│   ├── event-router/        # Normalization + routing + filters
│   ├── topic-registry/      # Topic CRUD + validation
│   ├── schema-registry/     # Schema versioning + compatibility
│   ├── metrics/             # Metrics collectors + dashboards
│   ├── database/            # Knex migrations + connection
│   └── sdk/                 # Client SDK
└── turbo.json               # Build pipeline config

Multi-Tenant Isolation

The platform supports multi-application tenancy:

Every API request carries an X-Application-Id header
Backend middleware extracts this into req.applicationId
All entity-owning APIs (topics, schemas, mappings, webhooks) filter by application
The Admin UI auto-attaches the current application ID from localStorage

Environment Promotion

Configuration (mappings, schemas, topics) supports a three-stage promotion workflow:

Development → Staging → Production

Each entity can be independently promoted between environments. The Environment Grid in the Admin UI provides a visual overview of sync status across all environments.

Dependency Graph

Turborepo manages the build dependency graph. Packages build in order of their dependencies:

shared-types
  └── shared-utils
       └── shared-config
            ├── observability
            ├── redis-layer
            ├── auth
            ├── event-router
            ├── topic-registry
            ├── schema-registry
            ├── metrics
            └── database
                 ├── backend (app)
                 ├── workers (app)
                 └── sdk

Performance Targets

Metric	Target
Write latency	< 50ms p95
Broadcast latency	< 100ms p95
Subscription latency	< 100ms
DB event latency	< 150ms