LM Studio 0.3.10: 🔮 Speculative Decoding

By LM Studio Team

• 2025-02-18

We're thrilled to introduce Speculative Decoding support in LM Studio's llama.cpp and MLX engines!

Speculative Decoding is a technique that can speed up token generation by up to 1.5x-3x in some cases.

Upgrade LM Studio to 0.3.10 via in-app update, or from https://lmstudio.ai/download.

Speculative Decoding

Speculative decoding is an inferencing optimization technique pioneered by works like Fast Inference from Transformers via Speculative Decoding by Leviathan et al., and Accelerating Large Language Model Decoding with Speculative Sampling by Chen et al. It can be viewed as a sort of a speculative execution optimization you might find in modern CPUs, but applied to LLMs inferencing.

In both LM Studio's llama.cpp and MLX engines, Speculative Decoding is implemented using a combination of 2 models: a larger LLM ("main model"), and a smaller / faster "draft model" (or "speculator"). The original llama.cpp implementation was authored by Georgi Gerganov, and MLX's by Benjamin Anderson and Awni Hannun. These implementations are continually improved upon by the open source community.

How it works

The draft model is being run first, quickly predicting the next few tokens as a "draft" generation. Right afterwards, the draft generation tokens are either confirmed or rejected by the main model. Only tokens that would have been generated by the main model are accepted. This results in potential speedups (when enough tokens are accepted) without any degradation in quality. In cases where tokens are rejected more often than not, you will likely see decreased total generation speed! The choice of models is important for achieving best results.

Performance Stats

LM Studio + MLX engine (Apple M3 Pro, 36GB RAM)

LM Studio + CUDA llama.cpp engine (NVIDIA RTX 3090 Ti 24GB VRAM, Intel Core Ultra 7 265K CPU, 32GB RAM)

Using Speculative Decoding in your chats

In LM Studio 0.3.10, you will find a new sidebar section for Speculative Decoding. Once you have a main model loaded (cmd/ctrl + L), you'll see compatible options for draft model in the new draft model selector.

Draft model compatibility

For a model to be used as a draft model for another model, they need to be "compatible". At a high level, for the draft model to be effective it must be able to generate from the same possible tokens the larger model would. Practically speaking, both models should share sufficiently similar vocabulary (the totality of tokens the model "knows" about) and tokenizer characteristics. LM Studio automatically checks whether your models are compatible for each other for the purposes of speculative decoding.

Visualize accepted draft tokens

Turn on accepted draft token visualization to see colorized tokens indicating whether they were generated from the draft model or the main model. The more green, the better.

Using Speculative Decoding via API

You can also use speculative decoding via LM Studio's local server. When enabled, you'll get back rich generation stats including new fields related to speculative decoding.

  "stats": {
    "tokens_per_second": 15.928616628606926,
    "time_to_first_token": 0.301,
    "generation_time": 1.382,
    "stop_reason": "stop",
    "draft_model": "lmstudio-community/llama-3.2-1b-instruct",
    "total_draft_tokens_count": 12,
    "accepted_draft_tokens_count": 10,
    "rejected_draft_tokens_count": 0,
    "ignored_draft_tokens_count": 2
  }

Option 1: Set a draft model in the Server UI

Similar to the chat, you can set a draft model in the config sidebar. Once set, requests targeting this model will leverage speculative decoding using the draft model you selected.

Option 2: draft_model in your request

You can also provide the draft model key as a part of your request payload. Learn more about JIT loading.

curl http://localhost:1234/api/v0/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "deepseek-r1-distill-qwen-7b",
    "draft_model": "deepseek-r1-distill-qwen-0.5b",
    "messages": [ ... ]
}'

How to use Speculative Decoding effectively

Read the documentation article: Speculative Decoding.

As a rule of thumb, aim to pair a large model with a smaller one. The draft model should be much smaller than the main model, and from the same family. For example, you can use Llama 3.2 1B as a draft model for Llama 3.1 8B.

Note: as of LM Studio 0.3.10, the system will try to offload the draft model entirely onto the GPU, if one exists. The main model GPU settings are configurable as usual.

When to expect worse performance

Two main factors will lead to worse performance: draft model too large compared with your available resources, and low draft acceptance rates. To avoid the former, always use draft models that are much smaller than the main model by comparison. The latter is model and prompt dependent. The best way to understand the possible tradeoffs of speculative decoding is to try it for yourself on the tasks you care about.

Finding compatible draft models

Since not all models are compatible with each other, it's important to identify models that can work together as a draft and main model.

An easy approach is using a large and small variant of the same model family, as illustrated in the table below:

Advanced Resources

We've compiled a few extra resources for those of you who want to dig deeper!

Papers:

• Fast Inference from Transformers via Speculative Decoding
• Accelerating Large Language Model Decoding with Speculative Sampling
• SpecInfer: Accelerating Large Language Model Serving with Tree-based Speculative Inference and Verification
• Accelerating LLM Inference with Staged Speculative Decoding

Compatibility check implementations

The following are excerpts from llama.cpp and MLX showing the current implementations for checking speculative decoding compatibility between models.

def is_draft_model_compatible(self, path: str | Path) -> bool:
    path = Path(path)
    draft_tokenizer = mlx_lm.tokenizer_utils.load_tokenizer(path)
    if draft_tokenizer.vocab_size != self.tokenizer.vocab_size:
        return False
    return True

Sources:

• llama.cpp: ggml-org/llama.cpp
• MLX: lmstudio-ai/mlx-engine