How We Built Real-Time AI Music Collaboration

The Problem

Live-coding music is inherently real-time. When an AI agent writes a new Strudel pattern, the musician in the browser needs to hear it immediately — not after a page refresh, not after a polling interval, but now. Building this required us to rethink how a traditional REST API delivers data.

StrudelHub connects AI agents to browser-based music sessions. An agent pushes Strudel code via the REST API, and the browser plays it using the Strudel audio engine. The challenge: how do you get code from a POST request on one server instance to an EventSource connection on potentially a different server instance, with sub-second latency?

The Architecture

Our stack is straightforward: a FastAPI backend, PostgreSQL for persistence, and a React SPA with the <strudel-editor> web component for audio playback. The real-time layer is where it gets interesting.

PostgreSQL LISTEN/NOTIFY

Instead of adding Redis or a message broker, we use PostgreSQL's built-in pub/sub system. When an agent pushes new code:

1. The API handler writes the code to the strudel_sessions table
2. It calls pg_notify('strudel_sessions', session_id) on the same transaction
3. All server instances listening on that channel receive the notification
4. Each instance checks its local SSE subscriber map and pushes the update

This gives us cross-instance fan-out with zero additional infrastructure. PostgreSQL is already our database — LISTEN/NOTIFY comes for free.

Server-Sent Events

We chose SSE over WebSockets for the browser connection. The reasons were practical:

• Simpler protocol — SSE is just HTTP with text/event-stream content type. No upgrade handshake, no frame parsing, no ping/pong.
• Automatic reconnection — The browser's EventSource API handles reconnection out of the box. If the connection drops, it reconnects and picks up where it left off.
• Cookie auth works — HTTP-only JWT cookies are sent automatically with EventSource connections. WebSockets would require a separate auth token mechanism.
• Load balancer friendly — SSE connections are standard HTTP, so ALB handles them without special configuration.

The Full Flow

Agent (HTTP POST /api/sessions/{id}/code)
  → FastAPI handler
  → Write to PostgreSQL (strudel_sessions table)
  → pg_notify('strudel_sessions', session_id)
  → All server instances receive NOTIFY
  → Each instance pushes to local SSE subscribers
  → Browser EventSource receives { type: "update" }
  → React app fetches latest code
  → <strudel-editor> evaluates and plays the pattern

The entire path from agent POST to audio playback takes under 200ms on a good connection.

Handling Multiple Instances

In production, StrudelHub runs on AWS ECS Fargate behind an Application Load Balancer. Multiple instances handle requests, and a user's SSE connection might land on a different instance than the one processing the agent's POST.

PostgreSQL LISTEN/NOTIFY solves this elegantly. Every instance maintains a dedicated asyncpg connection subscribed to the strudel_sessions channel. When a notification arrives, it carries only the session ID — each instance then checks its in-memory map of active SSE connections. If it has subscribers for that session, it pushes the event. If not, it ignores the notification.

This means:

• No sticky sessions required at the load balancer
• No shared state between instances (beyond PostgreSQL)
• Horizontal scaling is just adding more Fargate tasks

Preventing Feedback Loops

There's a subtle problem: when a user edits code in the browser and saves, the browser POSTs to /api/sessions/{id}/sync. If this triggered a NOTIFY, the same browser would receive its own update back via SSE and re-render — creating a feedback loop.

Our solution: browser sync writes to the database but does not call pg_notify. Only agent-initiated code changes trigger notifications. The browser already has the latest code (it just sent it), so there's no reason to echo it back.

What We'd Do Differently

If we were starting over, we'd consider using PostgreSQL's logical replication slots instead of LISTEN/NOTIFY for the notification mechanism. LISTEN/NOTIFY has a payload size limit (8000 bytes) and notifications are lost if no one is listening. For our use case — where the payload is just a UUID and instances are always connected — these limitations don't matter. But for a more general pub/sub system, logical replication would be more robust.

Try It Yourself

The best way to understand the real-time flow is to experience it. Create an account on StrudelHub, get an API key, and connect an AI agent. Watch the code appear in your browser as the agent writes it — and hear the music play in real time.

Check out our Getting Started guide to set up your first session, or read the API Reference to build your own integration.