AI Technical Due Diligence Template: What PMs Need to Evaluate Before Acquiring or Partnering
TL;DR
When your company evaluates an AI startup acquisition or a deep technical partnership, standard software due diligence misses the risks that actually matter in AI: model quality on your specific use cases (not benchmarks), data asset provenance and licensing exposure, inference cost at your production scale, and key-person risk concentrated in one or two applied scientists. This template gives AI PMs a structured checklist across six dimensions: model quality, data assets, engineering maturity, cost structure, intellectual property, and integration feasibility. Use it before signing a term sheet or an OEM agreement.
The AI PM Minute
One tactic to make you a sharper AI PM, twice a week. 60 seconds to read. Free.
No fluff. Unsubscribe anytime.
Why Standard Tech DD Fails for AI Companies
Standard technical due diligence examines code quality, architecture scalability, security posture, and engineering team depth. These still matter in AI deals. But the failure modes that have destroyed AI acquisitions almost never come from bad code. They come from four sources that most DD processes do not touch:
Benchmark inflation
A model that scores 92% on MMLU may score 47% on your actual use case. Public benchmarks are designed to be broadly meaningful, not specifically predictive of your task. Companies routinely present benchmark numbers that have no relationship to what you will actually experience.
Unlicensed training data
Many AI companies fine-tuned their models on scraped data with questionable provenance. As AI copyright litigation matures, this exposure can become a liability you inherit in an acquisition. Unlike traditional IP, you cannot audit what a model 'learned' from — you can only audit what data was used to train it.
Inference cost mismatch
A company that demos beautifully may be running at 10x higher inference cost than is commercially viable at your expected volume. Cost per query at 1,000 requests per day is meaningless; cost at 1 million matters enormously. Many AI startups have not run at scale and do not know their unit economics.
Key-person concentration
AI capability often sits in one or two applied scientists who designed the training pipeline, understand the quirks of the model, and hold the institutional knowledge of what works. If they leave post-acquisition, you may have the model weights but none of the knowledge to improve or maintain them.
Dimension 1: Model Quality Assessment
This is the core of AI technical DD and the area most companies handle worst. Your goal is to evaluate the model on tasks that represent your actual production use case, not the tasks the company chose to showcase.
Build your own evaluation set
Before DD begins, create 50 to 200 realistic examples from your production use case or target use case. Cover the full distribution of inputs, including edge cases and adversarial examples. This is your ground truth.
Run blind evaluations
Ask the target to run their model on your eval set without seeing your expected outputs. Have your team or a third party score the outputs against your criteria. Never accept their internal benchmark numbers as a substitute.
Test failure modes explicitly
Does the model hallucinate confidently on questions outside its knowledge cutoff? Does it refuse reasonable requests inappropriately? Does it degrade on inputs that differ slightly from its training distribution?
Check consistency across runs
Run the same 20 prompts three times at temperature 0. If outputs vary significantly, the model has fundamental output instability problems that will show up in production at scale.
Evaluate on your latency requirement
Measure p50, p90, and p99 latency under realistic concurrent load, not just sequential calls. Many AI products look fast in demos but buckle at production traffic levels.
Assess versioning discipline
How do they handle model updates? Do they version models? Do they have a process for evaluating regressions before deploying a new version? No versioning discipline means your product behavior can change without warning.
Red flags in model quality DD
- They refuse to run your eval set, citing IP concerns, without offering a sandboxed alternative
- Their only evidence of quality is third-party benchmark leaderboards
- They cannot answer what the model fails at
- Outputs are different every time for deterministic tasks
- No systematic evaluation pipeline exists; quality is assessed by "it felt good"
Dimension 2: Data Assets and Provenance
In traditional software acquisitions, code is the asset. In AI acquisitions, training data is often the most valuable and most legally exposed asset. These are the questions to ask:
Pre-training data
- What data was used to pre-train or fine-tune the model?
- Is it a base open-weight model (Llama, Mistral) or a proprietary pre-training run?
- If open-weight: what is the licensing of the base model for commercial use?
- If proprietary: what is the provenance of the training corpus? Was web-scraped data included?
Fine-tuning and instruction tuning data
- What datasets were used for fine-tuning? Are they licensed for commercial use?
- Was any data sourced from user interactions? Under what consent framework?
- Were third-party APIs (OpenAI, Google) used to generate synthetic training labels? (This violates most API terms of service and creates legal exposure.)
Proprietary data as a moat
- If they claim a data moat, what is the data? Who owns it?
- Are there exclusivity agreements with data providers?
- What happens to data exclusivity in an acquisition? Can a data provider terminate the agreement on change of control?
GDPR and CCPA exposure
- Does the training data include personal information about EU or California residents?
- Have they received any right-to-be-forgotten or data deletion requests? How do they handle them?
- Is there a documented data retention and deletion policy for training data?
Master the Business Side of AI Products
The AI PM Masterclass includes the full framework for evaluating AI vendor and partnership decisions, taught live by a Salesforce Sr. Director PM.
Dimension 3: Engineering and MLOps Maturity
AI engineering maturity is distinct from software engineering maturity. A team can write clean code and still have no systematic process for improving their model or diagnosing production failures.
Evaluation pipeline
Does an automated eval suite exist? How often does it run? Are there regression alerts before any model is deployed to production? A company with no eval pipeline is flying blind and will continue to do so post-acquisition.
Model versioning and rollback
Can the team roll back to a previous model version within an hour? If a model update breaks production at 2am, what is the incident response path? Many AI startups deploy model weights directly without version control.
Observability
What is logged per inference request? Can the team query for examples where the model failed? Can they reproduce a specific failure from a customer report? Observability is the foundation for continuous improvement.
Drift detection
How does the team know when model performance has degraded without a customer filing a support ticket? Systematic monitoring of input distribution shifts and output quality is table stakes for a mature AI engineering org.
Training reproducibility
Given the same data and config, can the team reproduce a trained model checkpoint? Many early AI teams have lost the ability to retrain because they did not track experiment configs and data versions systematically.
Dimensions 4 and 5: Cost Structure and Team Risk
Inference Cost at Your Scale
What is cost per 1K tokens or per inference call at their current scale?
What is the projected cost at 10x, 100x, and 1,000x their current volume?
Nonlinear scaling is common and dangerous.
Are they self-hosted or on a third-party inference provider?
Provider lock-in can mean cost surprises at volume.
What is GPU utilization during peak hours?
Low utilization means wasted cost; high utilization means latency pressure.
What are the GPU reservation commitments or spot instance dependencies?
Spot instances can be interrupted; reserved commits carry cost floor.
Team and Key-Person Risk
Who designed the training pipeline? Are they still with the company?
How many people can retrain the model from scratch if needed?
One person = critical risk.
What is the vesting schedule for the AI team? What happens post-acquisition?
Many offers include retention packages with cliffs.
Is there documentation of training configs, data processing, and hyperparameter choices?
Undocumented knowledge walks out the door.
What is the ratio of applied scientists to software engineers?
AI product engineering requires a different skill mix than traditional software.
Dimension 6: Integration Feasibility
The final dimension is whether the AI capability can actually be integrated into your product and your team in a reasonable time and at a reasonable cost. AI integrations consistently take 2 to 4x longer than expected because of invisible complexity in evaluation, prompt adaptation, and safety tuning.
API or model weights?
If you are acquiring model weights, you need inference infrastructure on your side. If it is an API, you have dependency risk but lower integration cost. Which is it, and is that what you actually want?
Customization requirements
Does your use case require additional fine-tuning? If yes, do you have the data and engineering capacity to run that process? Who supports you through the fine-tuning after the deal closes?
Safety and moderation alignment
Is the model's refusal behavior appropriate for your product context? Over-refusal is a real product problem in B2C; under-refusal is a real safety problem in many B2B contexts. Does the model's safety calibration match your use case?
Compliance requirements
Does your product serve regulated industries (healthcare, finance, legal)? Does the model meet the data residency, audit logging, and retention requirements those industries demand? Does the vendor have the compliance posture (SOC 2, HIPAA BAA) you need?
The PM role in technical DD
Technical DD in AI companies is too important to leave entirely to legal or corp dev. Product managers understand the use case requirements, can design the task-specific evaluation set, and can assess integration feasibility in context. If your company is doing an AI acquisition without a PM-led product assessment, you are likely to buy something that technically works but commercially fails.
Learn to Evaluate and Build AI Products That Win
The AI PM Masterclass teaches the full toolkit for evaluating, building, and managing AI products from concept to production.
Related Articles
Before you go: get the AI PM Minute
One tactic to make you a sharper AI PM, twice a week. 60 seconds to read. Free.
No fluff. Unsubscribe anytime.