Vision Language Models Explained for Product Managers
TL;DR
Vision language models (VLMs) are the architecture behind AI products that process images, documents, charts, and video. If your product accepts image input or generates image descriptions, you are shipping a VLM. This guide explains how VLMs work at the level that affects product decisions: how images become tokens, why visual inputs cost more than text, which architectural trade-offs determine capability limits, and what the current VLM landscape means for model selection.
The AI PM Minute
One tactic to make you a sharper AI PM, twice a week. 60 seconds to read. Free.
No fluff. Unsubscribe anytime.
What Vision Language Models Actually Are
A vision language model is an AI model that can process both images and text. It combines a vision encoder (a component that converts images into a numerical representation the model can reason about) with a language model (which generates text). The result is a model that can answer questions about images, describe visual content, read text in images, and reason across visual and textual information simultaneously.
VLMs are distinct from image generation models. Diffusion models like Stable Diffusion or Flux generate images from text descriptions. VLMs read images and produce text. Some frontier models (GPT-4o, Gemini 3.5) can do both, but the underlying mechanisms are separate architectural components.
Image understanding
Answering questions about a photo, describing what is happening in a scene, identifying objects. The foundational VLM use case.
Examples: Product defect detection, content moderation, photo-to-text for accessibility
Document and chart parsing
Extracting structured data from PDFs with tables, reading financial charts, parsing invoices and receipts. High-value enterprise use case because existing OCR misses structure.
Examples: Invoice processing, contract review, data extraction from research reports
Medical and scientific imaging
Analyzing X-rays, pathology slides, satellite imagery, molecular diagrams. Requires domain-specific fine-tuning on top of a general VLM.
Examples: Radiology assistance, drug discovery, environmental monitoring
Video understanding
Describing events across video frames, tracking actions, answering questions about video content. More expensive and still maturing.
Examples: Security monitoring, sports analytics, video search and retrieval
How VLMs See: Patches, ViT Encoders, and Visual Tokens
Before a VLM can reason about an image, it must convert that image into a sequence of vectors that the language model can process — the same format as text tokens. This conversion happens in three steps.
Step 1: Patch extraction
The image is divided into a grid of fixed-size patches, typically 16x16 or 14x14 pixels. A 512x512 pixel image divided into 16x16 patches produces 1,024 patches. Each patch is a small square region of the image that the model will process independently. Patches preserve local spatial information (what is in this region) before the model learns global relationships (how regions relate to each other).
Step 2: Vision Transformer (ViT) encoding
Each patch is flattened into a 1D vector and projected into a high-dimensional embedding space by the Vision Transformer, or ViT. Positional embeddings are added to tell the model where each patch appeared in the original image. The ViT then runs multiple attention operations across all patches, allowing each patch to incorporate information from its neighbors — capturing spatial context, edges, and scene structure.
Step 3: Projection into language model space
The visual patch embeddings live in the ViT's embedding space, which is not the same as the language model's token embedding space. A learned projection layer (often a small MLP or cross-attention mechanism) translates visual embeddings into the same dimensional space as text tokens. After projection, the model can process visual and textual tokens in a single sequence.
The token cost implication every PM needs to know
A 512x512 image at 16x16 patches produces approximately 1,024 visual tokens. A 1,024x1,024 image produces 4,096 tokens. Many providers add high-resolution tiling on top of this, producing 3,000 to 6,000 tokens per image at standard detail settings. Compare this to text: 1,024 tokens is roughly 750 words. Every image in your product adds the equivalent of a multi-paragraph text passage to your token count. This is why image-heavy AI features have dramatically different unit economics than text-only features. Always check how your provider bills image tokens before building pricing assumptions.
How VLMs Learn to See: Contrastive Training and Generative Alignment
A ViT encoder fresh from random initialization does not understand what a cat or a financial chart is. It needs to be trained to map image content to semantically meaningful representations. Modern VLMs train this understanding in two stages.
Stage 1: Contrastive pre-training (CLIP-style)
The vision encoder and a language encoder are trained simultaneously on hundreds of millions of image-text pairs from the internet. The training objective: make the image embedding and its matching text description as close as possible in the shared vector space, while pushing apart mismatched pairs. After this training, the vision encoder has learned to map images to the same semantic space as their text descriptions. 'A cat sitting on a red chair' as text and a photo of a cat on a red chair will produce nearby vectors.
Why it matters: This stage determines what concepts the model fundamentally understands. A VLM that was pre-trained without medical images will have a weaker semantic understanding of X-rays even after domain fine-tuning.
Stage 2: Generative alignment (instruction tuning)
The pre-trained vision encoder is connected to a language model and the system is fine-tuned on image-question-answer triplets. The model learns to generate text given an image and a question: 'What is in this image?' 'What does the chart show?' 'Read the text visible in this photo.' This stage adds instruction-following on top of visual understanding.
Why it matters: Most of the behavioral quality differences between VLMs — instruction following, refusal behavior, output format — come from this stage, not from the underlying ViT. Fine-tuning a good ViT with better instruction data produces a better product.
Learn to Make Better AI Model Decisions
The AI PM Masterclass covers how architectural decisions translate into product decisions — taught live by a Salesforce Sr. Director PM.
VLM Architecture Families and Their Trade-offs
Not all VLMs are built the same way. Three architectural families dominate the current landscape, each with distinct trade-offs for product use cases.
Encoder-only VLMs (CLIP, SigLIP)
Train a vision encoder to produce representations aligned with text, but do not include a generative language model. Output is an embedding vector, not text.
Strengths
Extremely fast. Cheap to run. Excellent for classification, retrieval, and similarity tasks.
Limitations
Cannot generate text descriptions. Cannot answer questions in natural language. Output is a vector, not human-readable.
Best for
Image search, content moderation classifiers, visual retrieval in RAG pipelines
Encoder-decoder VLMs (Flamingo, Idefics)
Combine a frozen or partially fine-tuned vision encoder with a decoder language model. Visual features from the encoder are fed into the decoder via cross-attention at each layer.
Strengths
Good at generating detailed, accurate descriptions. The cross-attention mechanism gives the language model rich access to visual features at every generation step.
Limitations
More complex to train and fine-tune. Cross-attention at every decoder layer is computationally expensive. Harder to scale to very large language models.
Best for
Image captioning, detailed visual question answering, scientific image analysis
Decoder-only VLMs (GPT-4o, Claude 3.5+, Gemini 3.5, LLaVA)
Project visual tokens directly into the language model's input space. The model processes visual and text tokens in a single sequence, with no architectural distinction between them.
Strengths
Simpler architecture. Benefits from all advances in language model scaling. Easier to extend to multiple modalities. Supports complex multi-step visual reasoning natively.
Limitations
Visual tokens are processed by the full attention mechanism, which is expensive at long contexts. Less interpretable than cross-attention architectures.
Best for
General-purpose visual reasoning, multi-turn visual conversations, complex document analysis, tasks requiring combined visual and textual reasoning
What VLMs Can and Cannot Do: The PM Decision Framework
Understanding the capability limits of VLMs prevents two common product errors: over-promising on visual AI features, and under-utilizing the genuine capabilities of current models. The following maps directly from the architectural constraints above.
Strong capabilities
Known failure modes
The tiling problem and high-resolution products
For images larger than the model's native resolution, providers split the image into overlapping tiles and process each independently before combining results. This increases token cost dramatically but also breaks spatial reasoning that spans tile boundaries. If your product relies on analyzing large-format documents (architectural drawings, full-page scans, medical imaging), test explicitly for tile boundary artifacts. Some use cases require image pre-processing (downsampling, cropping to regions of interest) rather than submitting the full high-resolution image.
The 2026 VLM Landscape for Product Decisions
The VLM market has consolidated significantly by mid-2026. A handful of frontier models dominate general-purpose use cases, while an increasingly capable open-weight tier serves latency-sensitive and data-sensitive applications.
GPT-4o and GPT-4o mini (OpenAI)
Frontier general purposeBest overall visual reasoning, especially on complex multi-step tasks. Strong OCR and document comprehension. GPT-4o mini offers a substantially cheaper option for simpler visual tasks.
Image tokens billed at a fixed rate per image tile. High-detail images can cost 5-10x low-detail.
Claude Sonnet 5 and Claude Opus 5 (Anthropic)
Frontier general purposeExcellent at following complex visual instructions, strong at chart and diagram interpretation, better calibration about visual uncertainty (more likely to say it cannot see something clearly rather than hallucinate).
Competitive with OpenAI on image pricing. Larger context window allows more images per call.
Gemini 3.5 Flash (Google)
Frontier economy tierStrong on structured document parsing, tables, and forms. Significantly cheaper than GPT-4o for document processing at scale. Native PDF processing (no need to convert to images).
Best cost-per-document for high-volume document processing use cases.
LLaVA family, Idefics, PaliGemma (open weight)
Open weight, self-hostedNo data leaves your infrastructure. Can be fine-tuned on proprietary domain data. Suitable for sensitive industries (healthcare, legal, finance). Lower ongoing cost at scale if infrastructure is already in place.
Higher fixed cost (GPU infrastructure), lower marginal cost. Break-even vs. API providers at roughly 1M to 5M images per month depending on model size.
The model selection heuristic for VLM-powered features: use the cheapest model that passes your product-specific eval. Do not default to the frontier model without benchmarking a cheaper alternative. Visual capability differences between frontier models and their economy tiers are small on routine tasks (OCR, simple scene description) and significant only on complex multi-step visual reasoning.
Build AI Products With Deep Technical Fluency
The AI PM Masterclass teaches the architectural concepts behind AI products so you can make better decisions about model selection, cost, and capability limits.
Related Articles
Before you go: get the AI PM Minute
One tactic to make you a sharper AI PM, twice a week. 60 seconds to read. Free.
No fluff. Unsubscribe anytime.