How to Run an AI Product Standup That Actually Surfaces Risk

When the model has degraded by 2 percentage points on the evaluation set, the engineer running the eval will not raise it as a blocker because no one is blocked. They will note it in a Slack thread or wait for the weekly review. By the time the PM hears about it, the team is two sprints into work built on top of a model that regressed. Standups must explicitly ask about quality movement, otherwise it stays invisible until it is large.

Tradeoff: Asking about quality every day adds about 2 minutes to standup. Skipping the question saves the time but loses the early warning. The teams that run weekly quality reviews instead of daily checks routinely catch drift only after it has compounded.

AI products generate more incidents per shipped feature than traditional software because model behavior changes under load and over time. The engineer who was on call last week may have been paged 8 times overnight and is exhausted today, but they will not say so unprompted. Standups that do not have a dedicated on call check in produce surprise resignations and silent quality decay because the on call is too tired to investigate root causes.

Tradeoff: A short on call check in feels redundant on quiet weeks. It is not. The 1 minute spent every Monday surfacing on call load is the cheapest health signal the team has.

A team building an AI feature without strong evaluation coverage on a relevant input slice will not call this out in standup, because there is no daily work item that says, fix the evaluation set. The gap shows up at acceptance review when the team realizes they cannot defend the quality of the release. Standups need a regular prompt about evaluation coverage so the gap is exposed weeks earlier.

Tradeoff: Asking about evaluation coverage is annoying for the engineering lead because the answer is usually, we need to do more work. That is the point. Surfacing the gap creates the work item that prevents the late stage scramble.

AI features almost always depend on other teams (platform, data, legal, applied science). When a dependency starts slipping, the dependent team often does not flag it because they hope to recover. The PM hears about the slip 2 sprints later when the date moves. A standup that explicitly asks about cross team dependencies surfaces these slips while they are small and recoverable.

Tradeoff: Listing dependencies every day duplicates information visible in roadmap tools. Yes, but the redundancy creates a forcing function for the PM to follow up before the slip becomes a date change.

The on call engineer reports overnight pages, current incidents, and any unresolved customer reports. Format, number of pages, top theme, hours of sleep impacted. If on call has been paged more than 5 times in the last week, the PM owns finding 1 hour of recovery time today (no meetings, no new work). This is non negotiable. Also surface any incidents in flight that the wider team should know about.

Tradeoff: Adding a recovery rule may slow short term progress because an engineer is offline. It prevents the larger cost, an exhausted engineer making a quality mistake or quitting.

The applied scientist or ML engineer reports any movement in the eval scores from the last 24 hours and flags any inputs that have started failing. Format, eval set name, score change, hypothesis for cause, planned investigation. If a metric moved more than 1 percentage point and no one knows why, that becomes the top investigation of the day. Skip this section only on days with no eval runs and announce that explicitly so the silence is not interpreted as good news.

Tradeoff: Daily eval reporting requires daily eval runs, which costs compute. The cost is small relative to the cost of catching a regression late. Set the eval to run automatically overnight so the morning report is ready before standup.

Each engineer reports against the three backlog streams (model work, bugs, features) with a focus on what is at risk, not what is on track. Use the prompt, what changed since yesterday and what is more or less likely to land. Avoid status theater. The PM listens for soft slip signals (the work is slightly bigger than I thought, the dependency has not responded) and parks them for postmeeting follow up.

Tradeoff: A risk first standup is harder for engineers because they have to surface what they are unsure about. Coach the team to do this for two weeks until it becomes habit. The early discomfort is worth the prevention of late surprises.

The PM names every cross team dependency the team is waiting on and the date they expect movement. If a date has slipped or the contact has gone quiet, the PM owns following up that day. Also surface decisions the team needs from outside (legal review, design approval, exec sign off). Decisions that have been waiting more than 5 business days get escalated by name in standup, which creates accountability for the PM to push.

Tradeoff: Naming slow decisions can feel confrontational. It is also the only reliable way to keep AI projects moving, because cross functional decisions are the most common source of multi week delays.