Event-Driven AI Architecture: Building Reactive AI Products with Async Patterns

Why Synchronous Architecture Breaks for AI

Traditional web APIs operate in milliseconds. A database read takes 5ms. A REST endpoint returns in under 100ms. HTTP connections time out after 30 to 60 seconds, and users abandon requests after 3 seconds. The entire assumption baked into synchronous request-response architecture is that the work is fast.

AI inference breaks every one of these assumptions. A Claude Opus 5 request generating a 2,000-token response takes 8 to 15 seconds. A GPT-5 call processing a 50-page document takes 20 to 45 seconds. A multi-agent pipeline with five sequential LLM calls can take 2 to 4 minutes. These numbers are not edge cases. They are typical.

HTTP timeout failures

Load balancers and API gateways typically kill connections after 30 to 60 seconds. A complex AI pipeline that takes 90 seconds returns a 504 Gateway Timeout instead of a result.

Poor user experience from blocking

Holding a user's browser waiting for a 20-second LLM call blocks the entire thread. Users see a frozen UI, assume the product broke, and abandon. The feature adoption rate drops even when the result is good.

Wasted infrastructure costs

Synchronous requests hold a web server process open for the full duration of the AI call. If you have 100 concurrent users waiting 15 seconds each, you need 100x more server capacity than a non-AI endpoint serving the same traffic.

No retry or recovery logic

When a synchronous AI call fails mid-request (model provider outage, rate limit, timeout), the user sees an error and the work is lost. There is no mechanism to retry automatically or resume from where it failed.

Coupling of concerns

In synchronous architecture, AI inference is tightly coupled to the user-facing request. Scaling inference requires scaling the entire web layer. Cost optimization at the model tier requires changes to the API layer.

Event-driven architecture solves all five by separating the act of submitting work from the act of receiving results. The user or calling system fires an event (start the AI task), and the result is delivered later through a separate channel.

The Four Core Patterns: Webhooks, Queues, Pub-Sub, and Polling

There are four primary patterns for building async AI products. They are not mutually exclusive. Most production AI systems use two or three in combination.

Pattern 1: Webhooks

How it works: The caller submits an AI job and provides a callback URL. When the AI processing completes, your backend sends an HTTP POST to that URL with the result.

When to use it: Best for B2B integrations where customers or partners build on your API. GitHub uses webhooks to notify external systems of events. Stripe uses them for payment status. OpenAI's Batch API uses them for async inference results.

PM note: Webhooks require the caller to run a server that can receive POST requests. This works for enterprise API customers but not for browser-based consumer apps. If your customers are consumers, pair webhooks with a polling endpoint or use WebSockets instead.

Pattern 2: Message Queues

How it works: AI tasks are placed into a queue (Redis, RabbitMQ, AWS SQS, Google Pub/Sub). Worker processes pull tasks from the queue, call the AI model, and store results in a database. The user polls or is notified when the job is complete.

When to use it: Best for internal AI workloads where you control both the producer (the web app) and the consumer (the worker). Document processing pipelines, batch analysis jobs, and background summarization all fit this pattern.

PM note: Queue depth is a leading metric to instrument. If the queue is consistently 200+ tasks deep, you either need more workers or you need rate-limit the inbound request rate. This metric predicts user-perceived latency before users complain.

Pattern 3: Pub-Sub (Publish-Subscribe)

How it works: Events are published to a topic. Multiple consumers can subscribe to the same topic and receive every event. Unlike a queue (where a task is consumed once), pub-sub delivers to all subscribers.

When to use it: Best when one AI event should trigger multiple downstream actions. A document upload might need to trigger: embedding generation, thumbnail creation, virus scan, and audit logging. With pub-sub, each service subscribes independently. Adding a new downstream service does not require changing the publisher.

PM note: Pub-sub is the right architecture for AI pipelines where the output feeds multiple systems. It enables you to add new AI-powered features (e.g., auto-tagging, sentiment analysis) to an existing event stream without touching the core upload flow.

Pattern 4: Long Polling and Server-Sent Events

How it works: The client submits an AI job, then immediately opens a second connection that stays open waiting for a result. With long polling, the client repeatedly asks 'is it done yet?' With Server-Sent Events (SSE), the server pushes updates as they arrive.

When to use it: Best for consumer-facing AI products where users watch a job complete in real time. ChatGPT's streaming response UI uses SSE. Status bars while an AI agent works through a task use long polling.

PM note: SSE with streaming token output is the gold standard for chat UX. It delivers the first token in under 1 second, making the product feel fast even when total generation takes 10 seconds. Streaming time-to-first-token is a KPI worth tracking separately from total generation time.

Designing the Right UX for Async AI Jobs

Architecture is only half the problem. Users do not know or care about queues. They care that they submitted a task and are not sure if it worked. Async AI UX design is about managing uncertainty and setting expectations.

Immediate acknowledgment

The user should receive confirmation within 200ms that their request was received and queued. 'Your document is being analyzed, we will notify you when it is ready' is the right pattern. Never leave a user staring at a spinner for 30 seconds without feedback.

Progress indication

For multi-stage AI pipelines, show intermediate progress. 'Extracting text (1 of 4)' is better than a static loading bar. Users who can see progress are 3x less likely to abandon while waiting.

Notification on completion

Browser notifications, in-app badges, email, or Slack DM. The right channel depends on expected wait time. Under 30 seconds: in-app update. 30 seconds to 5 minutes: banner notification. Over 5 minutes: email or Slack.

Job status endpoints

Build a job status API that returns pending, processing, completed, or failed with a progress percentage and estimated remaining time. This lets customers build their own integrations and lets your frontend poll gracefully.

Failure recovery UI

When an AI job fails, show exactly why and offer a one-click retry. 'The AI model was unavailable. Your document is in queue and will be retried automatically in 60 seconds' is far better than 'Something went wrong. Please try again.'

Result persistence

Async job results must be stored persistently. A user who closes their browser and returns 20 minutes later should see the completed result. Treating AI job results as ephemeral cache entries causes data loss and support tickets.

Build AI Products That Actually Scale

The AI PM Masterclass covers production architecture, async patterns, and the technical decisions that determine whether your AI product survives at scale. Taught live by a Salesforce Sr. Director PM.

Reliability Engineering for Async AI Pipelines

The hard part of async AI architecture is not building the happy path. It is handling failure gracefully across a pipeline where each step has a different failure mode.

Idempotency

AI jobs must be safe to retry. If a job fails and is requeued, running it twice should not produce duplicate outputs or double-charge the user. Each job needs a unique idempotency key so the system can detect and suppress duplicates.

Dead letter queues

Jobs that fail repeatedly should move to a dead letter queue rather than being retried indefinitely. This prevents a single bad job from blocking the entire queue and gives your team a place to investigate failures without losing data.

Backpressure

If workers cannot keep up with incoming jobs, the queue grows unboundedly. Implement backpressure by rejecting new jobs when queue depth exceeds a threshold, or by dynamically scaling workers. Letting queues grow to millions of items causes latency measured in hours, not seconds.

Circuit breakers

When a model provider is returning errors at 80%+ rate, stop sending requests immediately instead of saturating your queue and burning compute. A circuit breaker pattern opens automatically when error rates exceed a threshold and retries periodically to test if the provider has recovered.

Observability

Instrument every stage of the pipeline: job submission count, queue depth, worker throughput, per-stage latency, model provider error rate, and end-to-end job completion time. Without these metrics, diagnosing a slowdown is guesswork.

When to Choose Which Pattern: A Decision Framework

The right pattern depends on three factors: who the caller is (your app vs. a third-party customer), expected latency (seconds vs. minutes), and whether the result is point-in-time or streaming.

Consumer app, AI response in under 60 seconds

Use: SSE with streaming token output

Streaming feels fast even when total generation takes 15 to 30 seconds. Time-to-first-token under 1 second creates a perception of responsiveness. This is the ChatGPT pattern and it works.

Consumer app, AI job takes 1 to 10 minutes

Use: Background job queue with in-app push notification

Users cannot wait 5 minutes in the UI. Queue the job, let users navigate away, and send a browser push notification or in-app badge when complete. Include a 'results' page they can return to.

B2B API customer integrating your AI feature

Use: Async webhook with polling fallback

Enterprise customers expect webhooks. Provide a job ID they can poll as a fallback for customers whose infrastructure cannot receive webhooks (firewall restrictions are common). OpenAI's Batch API uses this model.

Internal AI pipeline with multiple downstream consumers

Use: Pub-sub with multiple subscriber services

One AI event (document upload, user action, data ingestion) often needs to trigger five different downstream processes. Pub-sub lets you add new consumers without touching the producer, which is the right abstraction for a growing AI product.

Batch processing of large datasets overnight

Use: Message queue with worker pool and priority lanes

Batch jobs should not compete with real-time user requests for worker capacity. Use a separate queue with lower priority and higher throughput workers optimized for sustained processing rather than low latency.

What AI PMs Need to Know to Influence Architecture Decisions

You do not need to implement these systems. You do need to know enough to push back, ask the right questions, and avoid shipping features with a fundamentally broken architecture.

Questions to ask your engineering team before shipping an AI feature:

1.What happens if the AI call takes 45 seconds? Does the user see a timeout error?
2.What happens if the model provider returns a 503? Is the job automatically retried, or is it lost?
3.Can we see how many jobs are in queue right now? What does that number look like under load?
4.If a user submits the same document twice, do we run the AI pipeline twice and charge twice?
5.Where are job results stored? For how long? What happens if the user comes back two days later?
6.How does this interact with rate limits? What happens when we hit 1,000 concurrent users?

The answers to these questions determine whether your AI feature is production-ready or a demo that falls apart under load. Event-driven architecture is not optional for AI products that need to scale. It is the foundation.