Local LLM outputs gibberish? Wrong chat template — not VRAM

The output is gibberish or it never stops talking

作者 Jakub Rusinowski

错误信息

<|im_start|>assistant<|im_end|> <|im_start|>user I I I the the of of<|endoftext|>assistantassistant

1. 1

   Pull from the official library instead of wiring the GGUF by hand最常见的修复方法

   Official Ollama library models (ollama pull llama3.1, qwen2.5, gemma3, …) ship with the correct template and stop tokens baked in — template gibberish essentially only happens with manually imported GGUFs and hand-written Modelfiles. If a hand-wired model misbehaves and the same model exists in the library, pulling the official version is the 30-second fix. Keep the manual route for models that genuinely aren't in the library (like your own fine-tunes).

   bash复制

   ollama rm my-broken-import
   ollama pull qwen2.5:7b   # template + stop tokens included

   生效了吗？Replies end cleanly with no tags, and ollama show <model> --template prints the family's template.

   检查哪些模型适配你的硬件find a model + quant that fits your hardware, with templates included打开显存检查器 →
2. 2

   Diagnose it: compare the template against the model family

   Confirm the mismatch instead of guessing. Every model family's correct template is visible on any properly configured model of that family, and on the model's Hugging Face card (tokenizer_config.json → chat_template). Thirty seconds of comparison tells you exactly which scaffolding your Modelfile should carry.

   bash复制

   # What template is my model actually using?
   ollama show my-model --template
   
   # What SHOULD a model of this family use?
   ollama show llama3.1 --template
   ollama show qwen2.5 --template

3. 3

   Fix the Modelfile: right TEMPLATE, right stop token

   When you must hand-write it (imported GGUF, converted fine-tune), copy the template from a same-family library model verbatim — never type it from memory; one missing newline or token matters. The stop parameter must match the family too: <|eot_id|> for Llama 3, <|im_end|> for ChatML/Qwen, <end_of_turn> for Gemma. Then recreate and retest.

   bash复制

   # Modelfile for a ChatML/Qwen-family GGUF
   FROM ./my-model.gguf
   TEMPLATE """<|im_start|>system
   {{ .System }}<|im_end|>
   <|im_start|>user
   {{ .Prompt }}<|im_end|>
   <|im_start|>assistant
   """
   PARAMETER stop <|im_end|>
   
   # apply:
   ollama create my-model-fixed -f Modelfile

   生效了吗？After ollama create, a short test prompt gets a single answer that stops by itself, with no tags and no invented user turn.
4. 4

   LM Studio: set the Prompt Format to match the model

   LM Studio reads the template embedded in GGUF metadata, and modern GGUFs carry the right one — but older or hand-converted files carry none, and LM Studio falls back to a guess. If output shows leaked tags, open the model's settings (My Models → gear icon → Prompt tab) and pick the correct format (ChatML for Qwen-family, Llama 3 for Llama-family, Gemma for Gemma) instead of the wrong autodetect. The moment tags stop appearing in output, you've got the right one.

   生效了吗？Replies stop leaking tags and end on their own after the change, across a few different prompts.
5. 5

   Fine-tune you converted yourself? Match the training template exactly

   A converted fine-tune must be served with the template it was trained on — which follows the base model family. This is the single most common reason a fine-tune that evaluated well 'comes out broken' in Ollama: the behavior is in there, the wrapper is hiding it. The GGUF conversion guide's Modelfile section walks the whole flow.

6. 6

   Template correct but still gibberish? Verify the file itself

   If the scaffolding checks out and output is still word salad, you're in the rarer failure class: a corrupted or truncated download, a broken experimental quant, or a GGUF too new/old for your runtime. Compare the file's size against the source repo listing (a partial download is smaller), re-download, prefer Q4_K_M-or-higher quants from reputable uploaders, and update Ollama/LM Studio — brand-new architectures need current runtime versions.

   bash复制

   # Size sanity check vs the Hugging Face repo listing
   ls -lh my-model.gguf
   
   # Update the runtime (new model support lands fast)
   ollama --version

   生效了吗？The file's size matches the repository listing exactly, or its SHA-256 matches the hash shown on the file's page on Hugging Face.