Feature Stores for AI Products: What Product Managers Need to Know
TL;DR
Feature stores solve a specific ML infrastructure problem that every PM building AI products at scale eventually collides with: training-serving skew. They ensure the features your model trained on are identical to what it sees at inference time. This guide covers what feature stores are, how Feast, Tecton, and Hopsworks work at a PM-relevant level, which product decisions depend on your feature store setup, and how the rise of LLMs is changing the calculus on when you need one at all.
The AI PM Minute
One tactic to make you a sharper AI PM, twice a week. 60 seconds to read. Free.
No fluff. Unsubscribe anytime.
The Training-Serving Skew Problem
Here is a failure mode that kills AI products quietly. The model works perfectly in evaluation, then underperforms in production. Accuracy drops. Recommendations feel off. The engineering team spends weeks debugging model code before someone finally checks whether the features fed to the model in production are the same ones it trained on.
They are not. This is training-serving skew, and it is the root cause of a large fraction of silent ML failures. A training pipeline computed a feature using historical data with one timestamp logic. The serving pipeline computing the same feature in real time used slightly different logic. Over time, the distribution drifted. The model was optimized for inputs it never sees.
Different computation logic
Training uses a batch SQL job. Serving uses a Python microservice. The same feature defined in two codebases will diverge. Timezone handling, null value treatment, aggregation windows — any difference corrupts the signal and degrades model accuracy in ways that look like model failure but are actually data failure.
Stale feature values
In training, all historical data is available at the correct timestamp. In serving, you compute features on demand. If a feature pipeline runs hourly but your model expects near-real-time freshness, every prediction is based on stale data that bears little resemblance to what training looked like.
Feature leakage
Training accidentally includes data that would not have been available at prediction time. The model looks significantly better in offline evaluation than it ever performs in production. Detecting and eliminating leakage requires careful timestamp discipline before training, not after.
Schema drift
A feature schema changes upstream: a field is renamed, an enum value is added, a null behavior changes. Training jobs cached the old schema. Serving pipelines read the new one. The model receives inputs it was never trained to handle, and it fails silently rather than loudly.
A feature store is the infrastructure that eliminates this class of problem by making features a first-class, shared, versioned artifact that training and serving both consume identically from a single source.
What a Feature Store Actually Is
A feature store is a data system that centralizes the definition, storage, and serving of ML features. It sits between your raw data sources and your model training and inference systems. The core principle: define a feature once, use it everywhere, guarantee consistency between training and production.
Registry
The catalog of all feature definitions: what each feature means, how it is computed, who owns it, and when it was last updated. This is the searchable source of truth that data scientists consult before building a new model.
Offline Store
Historical feature values optimized for training data retrieval. You query it with point-in-time correctness to reconstruct the exact features your model would have seen at any past timestamp, preventing leakage.
Online Store
Low-latency feature serving for real-time inference. Values are pre-materialized from batch pipelines or streamed from event systems so the serving path reads rather than computes on demand.
The critical mechanism is point-in-time correctness. When you generate training data, you specify the timestamp of each training example. The feature store retrieves the exact feature values that were available at that timestamp, preventing leakage and ensuring the model trains on realistic inputs that match what production will look like.
Why PMs need to understand this
Every time your ML team says "the model was great in testing but underperforms in production," training-serving skew is a probable root cause. Understanding feature stores lets you ask the right diagnostic question early: "Are training and serving consuming identical feature definitions from the same source?" This is not a technical detail you can delegate away — it determines whether your AI product's accuracy holds as you scale.
Key Platforms: What PMs Need to Know
Several feature store platforms have established themselves in the market. Each has a different philosophy and cost structure. You will encounter these in vendor evaluations, and understanding the tradeoffs helps you steer the decision toward the right fit for your team size and use case.
Feast (open source)
Powerful, configuration-heavy, no vendor lock-in
Strengths
Free, highly configurable, runs on your cloud infrastructure, large open source community. Strong for teams with ML engineering capacity who want full control over the stack.
Tradeoffs
Steep setup and ongoing maintenance burden. You are operating infrastructure, not buying a managed service. Expect significant ML engineering time upfront and continuously.
Best for
Teams with strong ML engineering capacity on a budget who need flexibility and do not want a third-party in their data path.
Tecton (managed SaaS)
Enterprise-grade, real-time feature pipelines with strong operational tooling
Strengths
Managed service eliminates infrastructure burden. Strong real-time streaming support for sub-second feature freshness. Monitoring, lineage, and access controls built in from day one.
Tradeoffs
Cost scales with data volume and can be significant at scale. Proprietary platform creates some level of lock-in. Requires procurement approval at many enterprise accounts.
Best for
Growth-stage and enterprise teams that need production-grade real-time features without dedicating a full ML engineering team to infrastructure.
Hopsworks
Full-stack ML platform with integrated feature store, model registry, and serving
Strengths
Integrated platform reduces tool sprawl. Can run on-premise or cloud, with a strong European data residency story. Feature store is one piece of a larger MLOps platform.
Tradeoffs
If you only need the feature store, you may be over-buying the full platform. Less mature than Tecton in some enterprise security and compliance areas.
Best for
Teams looking for an end-to-end ML platform, particularly those with on-premise requirements or strong European regulatory constraints around data localization.
Cloud-native options (Vertex AI Feature Store, SageMaker Feature Store)
Deeply integrated with your cloud provider, least friction if already committed
Strengths
No new vendor relationships. Tight integration with cloud ML training and serving infrastructure. Often included in existing enterprise agreements with Google or AWS.
Tradeoffs
Lags behind specialized vendors on advanced features like real-time streaming and monitoring depth. Creates deeper cloud vendor lock-in than a standalone solution.
Best for
Teams already standardized on a single cloud provider who want feature store as a configuration, not a separate infrastructure project.
Product Decisions That Depend on Your Feature Store
Feature store decisions are not just infrastructure decisions. They directly constrain what your product can and cannot do. These are the product implications that surface most often when a team builds or skips a feature store.
Real-time personalization
If your online store does not support sub-50ms feature serving, real-time personalization that updates within a session is not achievable. Batch-materialized features mean personalization reflects yesterday's behavior, not today's. The product UX of 'the AI knows me' depends entirely on feature freshness.
Feature reuse across models
A registry lets your second model reuse features your first model computed. Without one, each team recomputes the same features independently with subtle differences. This is both inefficient and a consistency risk that compounds as the number of models grows.
Rapid model iteration
With a feature store, a data scientist can pull any combination of registered features for a new experiment in hours. Without one, they spend days in data engineering before running a single baseline. Time to first experiment is the primary determinant of your AI team's iteration velocity.
Data governance and auditability
The registry makes features searchable and auditable. For regulated industries, this is not optional: you need to know what data feeds every model, when it was last refreshed, and who approved its use. Feature stores are the infrastructure layer that makes model governance possible.
Backfill and model retraining
When you retrain a model due to drift, new data, or a new feature, a feature store with point-in-time correctness makes generating clean training data straightforward. Without it, each backfill is a custom data engineering project that delays your retraining cycle by days or weeks.
A/B testing model variants
Running an experiment where model A and model B see different feature sets requires consistent, isolated feature serving. Without this, your experiment is confounded: you cannot attribute outcome differences to the model versus the features it consumed.
Go Deeper on AI Infrastructure in the Masterclass
Learn how ML infrastructure decisions constrain your product roadmap, taught live by a Salesforce Sr. Director PM.
How LLMs Change the Feature Store Calculus
The feature store conversation has shifted significantly with the dominance of LLMs. Traditional ML models consumed structured features: numerical scores, categorical labels, aggregated statistics. LLMs consume unstructured text and increasingly multimodal inputs assembled at inference time. This changes but does not eliminate the need for feature stores.
What changes with LLMs
When your model is a prompt-driven LLM, the inputs are the context you inject: retrieved documents, user history summaries, tool outputs, and system instructions assembled dynamically at inference time. RAG pipelines and vector databases handle much of what a feature store did for traditional ML. The offline/online store pattern is less directly applicable.
What stays the same
Most production AI products combine LLMs with traditional ML layers. A recommendation system might use a traditional ranking model to select candidates, then use an LLM to generate explanations. The ranking model still requires consistent, low-latency structured features. Feature stores remain essential for any non-LLM layer in a hybrid architecture.
The embedding store question
As embeddings become a core feature type (vector representations of users, items, and documents), some teams treat vector databases as a specialized feature store. Platforms like Tecton are adding native embedding support. The conceptual need (consistent, versioned entity representations) is the same; the implementation continues to evolve alongside vector infrastructure.
When you still need a traditional feature store
If your product uses structured predictive models alongside LLMs, processes tabular user or transaction data, requires sub-50ms inference latency on structured inputs, or operates under governance requirements that mandate feature auditability, a feature store remains necessary. LLMs eliminate the problem only for the parts of your product that are purely prompt-driven.
Do You Actually Need a Feature Store?
Not every AI product needs a feature store. Introducing one prematurely adds infrastructure complexity without solving a problem you actually have. The right time to introduce one is when the pain of not having it is costing more time than setup would require.
You likely need one if...
- ✓You have more than 2 ML models sharing features
- ✓Your models rely on real-time features updated within the session
- ✓Model accuracy consistently drops between offline eval and production
- ✓Data scientists spend more time in data engineering than modeling
- ✓You are in a regulated industry requiring feature audit trails
- ✓You are running A/B tests across model variants with different feature sets
You probably do not need one yet if...
- –Your product is primarily LLM-based with no traditional ML models
- –You have a single model with a small, stable feature set
- –Features are simple and computed at batch frequency (daily or weekly)
- –You have fewer than three data scientists on the team
- –You are pre-product/market fit and still validating the core AI use case
The signal to act is when a data scientist says "I spent two weeks just getting the training data right" or when your ML team reports that a retraining job failed because a feature schema changed upstream. That is the moment the feature store investment pays off fastest.
Turn ML Infrastructure Knowledge Into Better Product Decisions
The AI PM Masterclass teaches how infrastructure choices constrain your product roadmap and how to ask the right engineering questions before they become shipping blockers.
Related Articles
Before you go: get the AI PM Minute
One tactic to make you a sharper AI PM, twice a week. 60 seconds to read. Free.
No fluff. Unsubscribe anytime.