Model Versioning: Managing AI Models in Production

A code diff tells you exactly what changed: line 47 was modified, a new function was added, a dependency was updated. Model weights are billions of floating-point numbers. There's no meaningful way to 'diff' two model versions and understand what changed behaviorally. Two models with very similar weights can produce dramatically different outputs on edge cases. The only way to understand behavioral changes is through evaluation — running both versions against a test set and comparing outputs.

Trade-off: This means model versioning requires a robust evaluation pipeline, not just artifact storage. Without automated evaluation on every version change, you're deploying blind.

Code either works or throws an error. Models degrade gradually — accuracy drops by 2%, latency increases by 100ms, hallucination rate creeps up. These regressions don't trigger alerts unless you've explicitly built monitoring for them. A model that was excellent at launch can become mediocre over months as the data distribution shifts, and you won't know unless you're measuring continuously.

Trade-off: Model versioning must be coupled with continuous evaluation and monitoring. A version number without associated evaluation metrics is meaningless — you need to know not just which version is deployed, but how it's performing right now.

The same model with the same weights can produce different outputs for the same input due to temperature settings, sampling strategies, and floating-point arithmetic variations across hardware. This means you can't rely on snapshot testing (run input X, expect output Y) the way you would with deterministic code. Your evaluation must be statistical: does the distribution of outputs remain acceptable, not does every individual output match.

Trade-off: Evaluation sets need to be large enough for statistical significance. A 20-example test set isn't sufficient — you need hundreds of examples with multiple runs to detect regression with confidence.

A code bug typically affects a specific feature or endpoint. A model regression affects every interaction that uses that model. If your LLM powers search, summarization, and chat simultaneously, a single model update can degrade all three features at once. The blast radius of a bad model version is often much larger than a bad code deployment.

Trade-off: This is why gradual rollout is even more important for models than for code. Deploying a new model version to 100% of traffic immediately is the AI equivalent of pushing to production without testing.

The model itself — weights, architecture definition, and any adapter layers (LoRA, QLoRA). For self-hosted models, store the full checkpoint in a model registry (MLflow, Weights & Biases, Amazon SageMaker Model Registry). For API-based models (OpenAI, Anthropic), version the specific model identifier (e.g., 'gpt-4-0613' vs 'gpt-4-turbo-2024-04-09'). Pin to specific model versions in production — never use 'latest' or auto-updating model aliases.

Trade-off: Model checkpoints are large (gigabytes to terabytes). Storage costs add up when keeping many versions. Implement a retention policy: keep the last N versions readily available, archive older versions to cold storage, and delete versions older than your retention window.

The exact dataset used to train or fine-tune each model version. This includes training data, validation data, and any data preprocessing or filtering applied. Store a hash or snapshot ID, not a copy of the full dataset (which may be terabytes). Use data versioning tools like DVC, LakeFS, or Delta Lake to track dataset versions. Without data versioning, you can't reproduce a model, diagnose regressions, or prove compliance.

Trade-off: Data versioning adds pipeline complexity and storage overhead. For teams using third-party model APIs (not fine-tuning), version the prompt template and few-shot examples instead — these are your equivalent of 'training data' and have the same impact on output quality.

The performance metrics from running each model version against your evaluation set. Store accuracy, latency percentiles (p50, p95, p99), cost per request, and any domain-specific metrics (hallucination rate, format compliance, safety scores). Every model version should have an evaluation report that was generated before deployment, not after. These results are your decision record for why a model version was deployed or rejected.

Trade-off: Evaluation sets themselves evolve — new edge cases are discovered, requirements change. Version your evaluation sets alongside your models. A model that 'passed evaluation' on an outdated test set provides false confidence.

Temperature, top-p, max tokens, stop sequences, system prompt, tool definitions, and any post-processing logic. Two identical model weights with different temperature settings produce meaningfully different products. Version the full inference config as a JSON or YAML file alongside the model version. In many production systems, prompt changes are more frequent than model changes — and they have just as much impact on output quality.

Trade-off: Config changes are often treated casually ('just a prompt tweak') but can cause significant behavior shifts. Apply the same versioning and evaluation rigor to config changes as to model changes. A prompt change without re-evaluation is an untested deployment.