Small Language Models vs LLMs: How to Get Frontier Results from Open-Weight Models

A common starting point for an AI project is to pick the most capable model available and point it at every task. It works in a prototype. In production, it runs into three limits at once: cost, latency and control over data. The fix is to stop treating your AI system as one model and to give its narrow steps to small language models.

A small language model (SLM) is a compact language model, typically from tens of millions to around 10 billion parameters, built or trained to do narrow tasks such as classification, routing or extraction faster and cheaper than a large language model.

This post follows our post on open-weight models and asks the next question: once you have them, how do you combine small and large ones to get frontier-grade results?

What is a small language model?

A small language model is compact enough to run cheaply on modest hardware and specialized enough to do one job well. There is no fixed cutoff. Researchers at NVIDIA define an SLM as one that fits on a common consumer device and answers fast enough to serve one user's agent in practice, and note that, as of 2025, they would count most models below 10 billion parameters.

Most of these models don't write text. They output a label, a score or a yes/no answer. That's why they're so much lighter than a large language model (LLM).

"Small" still covers a wide range. The embedding model all-MiniLM-L6-v2 has about 23 million parameters, Qwen3-Embedding-0.6B about 600 million, and Qwen3-Reranker-8B about 8 billion.

Small models aren't new

Before generative models took over, most language AI was built from small, task-specific models, and deepset built many of them. Our question-answering model roberta-base-squad2 has about 125 million parameters and does one job: finding the passage that answers a question), and is still downloaded hundreds of thousands of times a month. Several generations of frontier models haven't displaced it, and that is the lesson: when a step is narrow and well defined, a small model that does it accurately and cheaply stays the sensible choice long after bigger models arrive.

Its distilled sibling, tinyroberta-squad2, runs at twice the speed with comparable quality.

What's changed isn't the idea. It's the combination: small models now work alongside LLMs in the same system, each doing the job it's best at.

Small language models vs LLMs: what's the difference?

SLM vs LLM comes down to scope: an LLM is a generalist that can reason, plan and write about almost anything, while a small language model does one narrow job at a fraction of the cost and latency.

• Size: SLMs run from tens of millions to around 10 billion parameters. Large open-weight LLMs run far larger: DeepSeek-V3 has 671 billion parameters in total, and Llama 3.1 comes in a 405-billion-parameter version.
• Output: most SLMs return a label, a score or a short span of text. LLMs generate open-ended text.
• Cost per call: an SLM is cheap enough to run on every request. An LLM needs more compute per call, so each call usually costs more, and agents multiply the number of calls: Anthropic reported that agents use about 4x more tokens than chat, and multi-agent systems about 15x more. If every one of those calls goes to the largest model, costs and response times multiply with them.
• Latency: SLMs answer fast enough to sit in front of every request. A larger model does more computation for every token it produces, so each LLM step usually takes longer.
• Where they run: SLMs fit on modest hardware you control. Large open-weight models need serious GPUs, and closed frontier models run on the provider's infrastructure.
• Where they fail: SLMs struggle outside the task they were trained for. LLMs work on repetitive tasks too, but you pay generalist prices and wait generalist times for each call.

That last point is the expensive one. As a post on enterprise AI costs put it, many companies are "using PhD-level models for intern-level tasks." Open-weight models solve the control problem; but they only solve cost and performance if you use them well.

When the LLM still wins

Use a large model where the task is open-ended, rare or low-volume:

• The task is open-ended. Complex reasoning, multi-step planning or creative synthesis across many sources.
• Requests are rare or unpredictable. Training a small model needs examples. If every request is different, there's no pattern to learn.
• Volume is low. Fine-tuning and hosting a model has upfront costs. As a rule of thumb, if a step runs only a few dozen times a day, a frontier model may well be cheaper overall.

Even the NVIDIA authors note that the economics are "highly case-specific" and that LLMs keep significant advantages on complex, multi-step tasks. The best systems are mixed: small models handle routine work, and large models take the hard cases.

What are examples of small language models?

Examples of small language models include Phi-4-mini, Qwen3-4B, Gemma 3 and SmolLM3 for text generation, ModernBERT for classification, and Qwen3-Embedding and BGE-M3 for search. They come in a handful of types, each matched to one kind of step in an AI system.

Which steps can a small language model handle?

Most of them. An AI application is a sequence of steps with different requirements, and usually only a few need a large model. A typical retrieval-and-answer flow:

1. Understand the request with a classifier: is it a question, a complaint or an order?
2. Decide where it goes with a router: simple lookup or complex analysis?
3. Find the right information with an embedding model and a ranker.
4. Draft the answer from the evidence, which needs a large model's reasoning.
5. Check the answer with a verifier: is it supported by the sources, and is it safe to send?

Only step 4 clearly needs a large model. The others are narrow, repetitive and well-defined.

Every AI system balances quality, cost and latency, and teams often treat that balance as fixed: if you want quality, you pay for a big, slow model. But if each step has different needs, the trade-off can be negotiated step by step.

An example: picture a workflow where one step sends every incoming document to a large model with the same prompt: "Classify this document as an invoice, contract, claim or other. Answer with one word." The prompt never changes, and the output is one of four labels. That step doesn't need the generative abilities of an LLM.

A small classifier trained on labeled examples of your documents could deliver the quality you need at a fraction of the cost, and run on infrastructure you control. Whether it does is something you validate per workflow.

The NVIDIA researchers estimated that in three popular open-source agent frameworks, roughly 40–70% of LLM calls could be handled by small models. They called small models "sufficiently powerful, inherently more suitable, and necessarily more economical" for many agent tasks. These are the authors' own estimates rather than measured results, but even the low end would mean four in ten calls could move off frontier pricing.

Small language model examples by type

Most are open-weight models on Hugging Face, and the MTEB leaderboard compares embedding models independently.

Check licenses before you choose. Open weights don't always mean commercial use: jina-embeddings-v3, for example, is released under CC BY-NC 4.0, a non-commercial license.

What is harness engineering, and where does RAG fit?

Harness engineering is "the discipline of designing the systems, constraints, and feedback loops that wrap around an AI model to make it reliable in production". RAG is one part of that harness: the part that gives a model the facts it needs. Small language models can sit in almost every part of it, and one part, routing, decides which model handles which step.

The parts of a harness, and where small models sit

• Context engineering chooses what the model sees at each step: instructions, examples, history and tool definitions. A small generative model can rewrite a vague query before search.
• Retrieval (RAG). Retrieval-augmented generation searches your documents at query time and adds the best passages to the prompt, so the model answers from your data without retraining. A study comparing retrieval with fine-tuning found that retrieval consistently did better at adding knowledge, for both information the model had seen before and entirely new information. The search itself runs on small models: an embedding model finds candidates, and a ranker picks the best.
• Routing decides which path, and which model, handles each request. This is LLM routing, and the router is usually a small classifier.
• Checks and verification test each output before it moves on: is it grounded, safe and in the right format? Groundedness and safety verifiers are small models, and a failed check can trigger a retry.
• Tools and memory let the system act and remember: tools call search, APIs or other pipelines, and memory stores facts across sessions. Choosing the next tool is open-ended reasoning, so the large model usually keeps this job.

Which lever should you pull first?

Think of the levers as a ladder, least effort first:

1. Better instructions and context. Context engineering chooses what goes into the prompt: clear instructions, good examples and the right data. It's cheap, though longer prompts add some cost and latency.
2. Retrieval. If the model gets facts wrong, give it the facts at query time.
3. Routing and checks. Break the task into steps, route requests to the right path, and add verification and retries.
4. A small model swapped in. Once the steps are clear, give the narrow ones to small models like those in the table above.
5. Fine-tuning. If a small model is close but not quite good enough, train it further on examples of your task. Fine-tuning is one kind of post-training, the umbrella term for all training after a model's first build. Distillation, covered next, uses a large model's outputs as that training data.

It's tempting to jump straight to lever 5. Start lower: levers 1–3 are cheaper, and they make levers 4 and 5 easier because the task is already well defined.

How can you customize a small language model: fine-tuning, distillation or both?

You customize a small language model by training it further on examples of your task, either labeled by you (fine-tuning) or produced by a larger model (distillation). Each option changes something different:

• Prompting and context change what the model sees; the weights stay the same.
• Retrieval changes what the model knows at query time; the weights stay the same.
• Fine-tuning changes the weights, using examples of inputs and the outputs you want.
• Post-training is the umbrella term for all training after a model's first build, fine-tuning included.
• Distillation is fine-tuning in which a large "teacher" model produces the training data.

The first two are harness changes. The last three change the model itself: the top rung of the ladder.

Fine-tuning vs RAG: when is fine-tuning worth it?

Fine-tune when the model has the facts but keeps getting the task wrong; use retrieval when it is missing the facts. A useful rule of thumb: retrieval gives a model knowledge; fine-tuning teaches it habits.

Fine-tuning is worth it when the model keeps getting the format, tone or behavior wrong, or when you run the same narrow task at high volume. It isn't worth it for missing or changing facts: baking facts into a model means retraining every time they change.

It is also more accessible than it was. LoRA (low-rank adaptation) trains a small add-on instead of all the model's parameters, and QLoRA showed a 65-billion-parameter model fine-tuned on a single 48GB GPU. For a step-by-step walkthrough, see our guide on how to fine-tune an LLM.

What is model distillation?

Model distillation is training a small "student" model on the outputs of a large "teacher" model, so the student learns to do the teacher's task at a fraction of the size. The idea was formalized by Hinton, Vinyals and Dean, and it is the most direct route from a working LLM prototype to a small language model in production.

DistilBERT cut BERT's size by 40% while keeping 97% of its language understanding and running 60% faster. Google researchers showed a 770-million-parameter model trained this way outperforming a 540-billion-parameter model on a benchmark task, over 700 times its size.

The NVIDIA researchers describe a practical recipe for agents:

1. Log your agent's model calls.
2. Group them into recurring tasks.
3. Pick a small model for each task.
4. Fine-tune it on the logged data.

Two cautions:

• Check the terms of your teacher model. OpenAI's terms, for example, say you can't "use Output to develop models that compete with OpenAI". Using an open-weight model as the teacher avoids the issue.
• Fine-tuned models don't carry over. A fine-tuned model, or a LoRA adapter, is tied to the base model it was trained on, so when you move to a newer base model you train again. What does carry over is your training data, your test set and your harness. Treat those as the assets, and the model as a replaceable part.

How do you know a small language model is good enough?

You test it against your own task before any production traffic depends on it. Build a test set from your real task: real inputs, with the outputs you expect, including the awkward edge cases. A few hundred well-chosen examples is usually enough to start, and you grow the set over time.

The test set does three jobs:

1. It tells you whether a smaller model is good enough for a given step.
2. It shows you which rung of the ladder helped. Did better retrieval fix it, or did you need fine-tuning?
3. It lets you swap models safely. When a better or cheaper open-weight model is released, run it against the same test set. If it passes, swap it in.

A test set only works if the rest of the system stays stable around the models you swap. That stable layer is the harness, and it is what Haystack and Haystack Enterprise Platform are built for.

How do you use small language models and LLMs to build a production-grade AI system with Haystack?

You build the harness as a pipeline in which each step is a component with its own model. Haystack, deepset's open source framework, has a component for each part of the harness:

• Context: ChatPromptBuilder assembles instructions, history and retrieved documents into the prompt.
• Retrieval: retrievers fetch documents, and embedders and rankers can run small models locally.
• Routing: routers such as ConditionalRouter, which routes on conditions you define, and TransformersTextRouter, which routes on a small classifier's label.
• Checks: pipelines can loop with a set repetition limit, so a validator can send an answer back to the generator for correction.
• Tools and memory: the Agent component lets an LLM call tools in a loop, keeping state across calls, until an exit condition is met.

Haystack Enterprise Platform, built on Haystack, adds a visual builder, managed deployment, observability, access control and enterprise deployment options. Its indexes prepare your files for retrieval once and are reused across pipelines, and an agent's tools can be pipelines, Python functions, MCP servers or Mem0 memory.

The platform is model-agnostic: you can use models from OpenAI, Anthropic, Google, Amazon Bedrock, Mistral, NVIDIA and others, connect Hugging Face models, or use models you host yourself, and reuse custom model definitions across pipelines. You can self-host the platform in a private cloud, on premises or air-gapped, so files, document stores and models stay in infrastructure you control. In air-gapped setups you host your own LLMs, embedding and ranking models, which is where small models pay off.

To measure, Pipeline Evaluation (in beta) scores real sessions with code checks or LLM-as-a-judge metrics, and a Candidate and Baseline Experiment runs a new pipeline version, such as one with a smaller model, against the current one on the same test cases. The AI Operator takes a service published through Gateway (both in beta), measures its cost and quality against an Experiment you define, and suggests cheaper configurations that keep that quality; you decide whether to apply them.

What's next for small language models?

The models will keep changing; the way you build around them doesn't have to. The releases in our open-weight models post will be overtaken, and so will the small models above.

That is why the test set matters more than any model. Teams with a stable harness, a stable test set and swappable models can adopt each improvement as it arrives instead of rebuilding.

You don't need a frontier model for every step. An AI system is a chain of steps, so quality, cost and latency can be set step by step: small language models take the narrow, repeated work of classifying, routing, ranking and checking, and large models take the open-ended reasoning. Pull the cheap levers first, and measure every change against your own tasks.

FAQ

What is LLM routing?

LLM routing sends each request to the model best suited to it, for example simple questions to a small model, hard ones to a large model and off-topic ones to a polite decline. The router is usually a small model itself, such as Arch-Router or a classifier trained on your own routes.

Is a small language model cheaper than an LLM?

Per call, almost always, because it is smaller and often runs on cheaper hardware. Overall, it depends on volume: fine-tuning and hosting have upfront costs, so for a step that runs rarely, a frontier model can still be cheaper.

Are small language models open source?

Many are released as open-weight models on Hugging Face, but licenses vary. Some, like jina-embeddings-v3, are non-commercial, so check each license before you use a model in production.

Can a small language model replace an LLM entirely?

Rarely. Small models are strong on narrow, repeated tasks and weak outside them, so most production systems combine both: small models for routine steps and a large model for open-ended reasoning.