NVIDIA Nemotron 3.5 Lightning for Product Managers

Most AI products send every request to a single model, whether it is a frontier model or a workhorse model. Nemotron 3.5 Lightning is designed to break that assumption. It is a 30B parameter model built on a hybrid Mamba-2, MoE, and Attention architecture, but only 3B of those parameters are active per token. This is the MoE design in action: the model routes each token to a specialized subset of its parameter space rather than activating everything at once.

NVIDIA trained Lightning by distilling the larger Nemotron 3 Ultra model into a faster, leaner package. The result is a model that punches above its weight on the specific tasks agentic systems do constantly: tool calls, document retrieval, output validation, data formatting, classification, and multi-step summarization. For those tasks it is not a frontier model, but it is fast enough and capable enough that sending those requests to GPT-5 or Claude Opus is simply expensive overkill.

NVIDIA released Nemotron 3.5 Lightning alongside NeMo Switchyard, an open-source model router. The two are designed to work together. Switchyard sits between your application and a pool of models. For each incoming request, it decides whether to route to a lightweight model like Lightning or escalate to a frontier model, based on estimated task complexity, latency budget, cost ceiling, and accuracy requirements you configure.

The key claim from NVIDIA benchmarks: NeMo Switchyard cuts inference costs by 74% compared to routing all traffic through frontier-only models, with a 6% accuracy drop. Whether that tradeoff works for your product depends entirely on your accuracy tolerance and your cost structure. NVIDIA reports that internal benchmarks show Switchyard maintaining frontier-level accuracy while cutting task completion cost to roughly one-third of Opus 4.8 alone.

The biggest insight behind Nemotron 3.5 Lightning is the distinction between the orchestration layer and the execution layer in an agentic system. The orchestration layer is where complex reasoning happens: understanding the user goal, decomposing it into subtasks, deciding which tools to invoke, and synthesizing results. The execution layer is where those subtasks actually run: querying a database, formatting a report, extracting entities from a document, calling an API, and validating the output.

Most agent systems today use a single frontier model for both layers. Lightning is designed specifically for the execution layer. The bet is that most of what an agent actually does, in terms of token volume and call frequency, is execution-layer work that does not require frontier-level reasoning. If that is true for your product, Lightning can absorb a large share of your inference spend at a fraction of the cost.

NVIDIA's 74% cost reduction claim deserves scrutiny. The benchmark compares routing all traffic through frontier models versus using Switchyard to split execution-layer traffic to Lightning. Whether your product sees savings anywhere near that depends on three variables: what share of your token volume is execution-layer work (classifiable and well-defined), your tolerance for the 6% accuracy drop on those tasks, and your current per-token price with your frontier provider.

The evaluation path for Nemotron 3.5 Lightning is different from evaluating a frontier model. You are not asking whether it can do everything your current model does. You are asking whether it can handle a specific slice of your workload without degrading user outcomes. That requires a workload audit before you touch any code.

This is the decision tree to run before committing engineering time to Lightning integration.