Gemma 4 12B: Download, Ollama, GGUF & QAT Setup Guide

Gemma 4 12B Unified landed on June 3, 2026. One weights file handles text, images, and audio without bolting on separate vision and audio encoders like the smaller E2B/E4B builds. 11.95B parameters, Apache 2.0, and a Q4 Ollama pull around 7.6 GB makes it the model people actually try on a 16 GB laptop. On June 5, Google shipped Quantization-Aware Training (QAT) checkpoints for the entire Gemma 4 family — the 12B Q4_0 GGUF now weighs about 6.7 GB with better quality than standard PTQ. Setup paths below include the new QAT repos.

This page is install notes: what changed from Gemma 3, how folks wire it into OpenCode, Codex CLI, and Crush, and where it still loses to cloud agents like Claude Code. For the wider 2026 model map, see Latest AI Models Compared (2026). Install paths start in Gemma 4 12B local setup.

Last updated: June 8, 2026. Live on aitoolsradar.org.

Quick specs

What makes Gemma 4 12B different

Google calls this variant Unified because it removed the standalone vision and audio encoders used on Gemma 4 E2B, E4B, and 31B. Instead:

• Vision: 48×48 pixel patches project through a small embedder (~35M params) into the decoder.
• Audio: 16 kHz waveforms become 40 ms frames, then a linear projection into the same embedding space.
• Text: Standard decoder-only transformer with hybrid attention (sliding window + global layers; final layer always global).

You skip the extra 150M to 550M vision tower and 300M audio encoder files that other Gemma 4 sizes carry. One download, one runtime.

Unified does not mean light on RAM. KV cache, projector weights, and image token budgets (70 to 1120 tokens per image on the card) still eat VRAM.

Benchmarks people cite (instruction-tuned, vendor table)

Google published these instruction-tuned numbers on the model card. We did not rerun the suites. Good for ballpark comparisons, not for picking a winner on your repo.

For coding, stare at LiveCodeBench and Codeforces. For screenshots and PDFs rendered as images, add MMMU Pro. The table compares Gemma 4 12B to Gemma 3 27B, not Gemma 3 12B.

How people use it (launch week, June 2026)

From HN, Reddit, and a few local runs we trust:

1. Local coding and 2026 agent CLIs

• llama.cpp / Ollama + Q4: Developers pull gemma4:12b and run vibe-coding tests. Senko’s minesweeper bench (linked from HN) reported ~5 tok/s on a 12 GB GPU with fixable syntax mistakes, not frontier-cloud quality, but usable offline.
• Terminal agents (June 2026): Builders wire the same local endpoint into OpenCode, Codex CLI (ollama / lmstudio providers), Crush, or Pi via baseURL in config, not legacy IDE-only chat extensions.
• Cloud agents (different lane): Claude Code, Cursor Agent, Google Antigravity, and GitHub Copilot cloud agent are common for paid daily coding in 2026. They do not natively target Ollama; you keep them on Anthropic/OpenAI/Google models and use Gemma 12B for private local passes.
• LiteRT-LM serve: Google documents port 9379 as an OpenAI-compatible shim for local Gemma (Edge blog). Point any agent that accepts a custom baseURL at that server.
• Hybrid stacks: HN commenters often pair Gemma 4 for multimodal with Qwen 3.5 9B for coding on the same 16 GB box, or run Codex/Claude Code in the cloud and OpenCode + Ollama offline on the same repo.

Worth the disk space if you want offline screenshots, audio clips, or drafts you will not send to a cloud API. It will not replace Claude Code or Codex on a nasty refactor. For terminal work, OpenCode or Crush pointed at Ollama is the usual pattern.

2. Multimodal workflows (OCR, PDF, screenshots, video, ASR)

3. On-device (Mac and iPhone)

• Google AI Edge Gallery on iOS and macOS for on-device Gemma family models.
• Eloquent for voice edit with 12B on Mac (June 2026 launch week).
• LiteRT-LM for cross-platform CLI import and litert-lm serve.

Hardware still matters: 12B on a phone is not guaranteed on every device.

4. Fine-tuning

• Unsloth Gemma 4 train guide: unified LoRA touches vision, audio, and text in one pass.
• Show HN: gemma-tuner-multimodal for Apple Silicon audio fine-tunes. Authors warn about OOM near ~2k tokens on 64 GB Macs.

Gemma 4 12B vs Gemma 3 12B (and vs E4B)

Gemma 4 12B local setup (June 2026)

Pick one path below. Default checkpoint: google/gemma-4-12B-it unless you host the base model yourself.

Path A: Ollama (fastest for most people)

Best when you want gemma 4 12b ollama in one command.

ollama pull gemma4:12b
ollama run gemma4:12b

Sampling (vendor readme): temperature 1.0, top_p 0.95, top_k 64.

Thinking in Ollama: put <|think|> at the start of the system prompt.

Vision prompt order: image before text.

Audio on 12B: not on Ollama’s 12b tag as of June 5. Use Path B or D for audio, or gemma4:e4b in Ollama for smaller audio-capable builds.

Path B: llama.cpp + Unsloth GGUF (control + vision)

Best when you care about gemma 4 12b unsloth quant quality. Unsloth now ships QAT GGUFs alongside the original quants.

1. Download from unsloth/gemma-4-12b-it-GGUF or the QAT variants at unsloth/gemma-4-12B-it-qat-GGUF.
2. Grab mmproj-BF16.gguf (~175 MB) for vision.
3. Recommended quant: UD-Q4_K_XL (~7.37 GB) per Unsloth KL benchmarks.

./llama-server \
  --model gemma-4-12b-it-UD-Q4_K_XL.gguf \
  --mmproj mmproj-BF16.gguf \
  --temp 1.0 --top-p 0.95 --top-k 64 \
  --port 8001 \
  --chat-template-kwargs '{"enable_thinking":false}'

GGUF size reference (12B IT, weights only):

Path C: MLX on Mac (gemma 4 12b on mac)

Unsloth ships an install script for MLX chat:

curl -fsSL https://raw.githubusercontent.com/unslothai/unsloth/refs/heads/main/scripts/install_gemma4_mlx.sh | sh
source ~/.unsloth/unsloth_gemma4_mlx/bin/activate
python -m mlx_vlm.chat --model mlx-community/gemma-4-12B-it-4bit

M2/M3 16 GB: workable for Q4-class 12B with modest context. M2 8 GB: prefer gemma4:e4b or heavy quants.

Path D: Transformers (full multimodal + audio)

Best when you need the official template for audio URLs, video frames, and enable_thinking.

pip install -U "transformers>=5.10.1" torch accelerate torchvision librosa

from transformers import AutoProcessor, AutoModelForMultimodalLM

MODEL_ID = "google/gemma-4-12B-it"
processor = AutoProcessor.from_pretrained(MODEL_ID)
model = AutoModelForMultimodalLM.from_pretrained(
    MODEL_ID, dtype="auto", device_map="auto"
)

messages = [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "Summarize this repo layout in five bullets."},
]

inputs = processor.apply_chat_template(
    messages,
    tokenize=True,
    return_dict=True,
    return_tensors="pt",
    add_generation_prompt=True,
    enable_thinking=False,
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=1024)

For images, put {"type": "image", "url": "..."} before text. For audio, put {"type": "audio", "audio": "..."} after instruction text (per model card).

Path E: LiteRT-LM (gemma 4 12b litertlm)

Google’s path for OpenAI-compatible local API and Mac agent workflows:

litert-lm import --from-huggingface-repo=litert-community/gemma-4-12B-it-litert-lm \
  gemma-4-12B-it.litertlm gemma4-12b
litert-lm serve

Wire 2026 agents to http://localhost:9379/v1 (verify port in your build):

Not a native local target: Claude Code, Cursor (BYOK cloud APIs only), Antigravity and Antigravity CLI (cloud models; replaces deprecated Gemini CLI).

Path F: vLLM (production GPU)

Use vLLM Gemma 4 12B recipe with nightly / gemma4-unified images. Plan 40 GB+ VRAM for comfortable BF16 serving; for lower VRAM, use the QAT compressed-tensors checkpoint at google/gemma-4-12B-it-qat-w4a16-ct (152k+ downloads). SGLang also supports the QAT checkpoints for efficient serving.

vllm serve google/gemma-4-12B-it \
  --max-model-len 16384 \
  --enable-auto-tool-choice \
  --reasoning-parser gemma4 \
  --tool-call-parser gemma4 \
  --limit-mm-per-prompt '{"image": 4, "audio": 1}'

VRAM and RAM planning

Google’s weight-only table (verify-live on your quant):

Real-world rule: add 2–8+ GB for KV cache depending on context. Gemma models with large vocab can blow past weight size at 32K context (Ollama community reports).

KV cache tip (Ollama): try OLLAMA_KV_CACHE_TYPE=q8_0 if context spikes VRAM.

Gemma 4 QAT (Quantization-Aware Training) — June 5, 2026

Two days after the 12B launch, Google released QAT checkpoints for every Gemma 4 size. Instead of quantizing after training (PTQ), QAT simulates quantization during training itself. The result: compressed weights that hold more of the original model quality than standard PTQ quants.

For 12B, you now have three new Hugging Face repos:

Unsloth also ships QAT GGUFs at unsloth/gemma-4-12B-it-qat-GGUF (121k+ downloads) with their UD quants. And Google published MTP QAT checkpoints so you keep the multi-token prediction speedup even with quantized weights.

Which quant to pick now: the QAT Q4_0 at ~6.7 GB is roughly 0.4 GB smaller than the standard Q4_K_M and Google’s benchmarks show it holds quality closer to the original BF16 model. If you already pulled gemma4:12b in Ollama and it works, no need to switch immediately. But for fresh installs or vLLM serving, start with the QAT checkpoints. You can also run QAT models directly in the browser via Transformers.js.

Thinking mode, tools, and system prompts

Gemma 4 adds a native system role and structured function calling tokens. For agents:

1. Declare tools in apply_chat_template(..., tools=[...]).
2. Enable thinking when plans are hard: enable_thinking=True.
3. On the next user turn, drop thought channels from history, except between tool calls in one agent turn (thinking docs).

Multi-token prediction (MTP): optional drafter checkpoints for faster inference (MTP overview). Supported stacks include Ollama, MLX, vLLM, and LiteRT-LM. Google also released MTP QAT checkpoints, so you can combine faster decoding with the quality-preserving QAT quantization.

Coding agents in 2026 (what to pair with Gemma 4 12B)

If you are picking agents in mid-2026, split the problem in two: terminal CLIs that can hit a local OpenAI URL, and cloud IDE agents that want Anthropic, OpenAI, or Google keys. Gemma 12B only fits the first bucket.

What developers actually run (June 2026)

Google’s launch blog still name-drops older OpenAI-shim tools. In June 2026 threads the local wiring is usually OpenCode, Codex CLI, or Crush.

Recommended stack: local Gemma + cloud frontier

1. Run weights: ollama pull gemma4:12b or litert-lm serve.
2. Local agent: ollama launch opencode or Pi/Codex pointed at http://localhost:11434/v1.
3. Hard tasks: same repo in Claude Code (Opus) or Codex (GPT-5.5 class) on cloud.
4. Google UI lane: Antigravity or Antigravity CLI. Google is retiring the consumer Gemini CLI in favor of Antigravity CLI; enterprise Gemini CLI continues. Local Gemma stays on Ollama/OpenCode.

OpenCode + Ollama (copy-paste pattern)

{
  "$schema": "https://opencode.ai/config.json",
  "provider": {
    "ollama": {
      "npm": "@ai-sdk/openai-compatible",
      "name": "Ollama (local)",
      "options": { "baseURL": "http://localhost:11434/v1" },
      "models": {
        "gemma4:12b": { "name": "Gemma 4 12B (local)" }
      }
    }
  }
}

Docs: Ollama + OpenCode. Start around 8k to 16k context on a 16 GB box. Raise num_ctx until VRAM complains; 64k is wishful on most laptops.

Codex CLI + Ollama (sketch)

In ~/.codex/config.toml (verify-live against advanced config):

[model_providers.local_ollama]
base_url = "http://localhost:11434/v1"

Then select gemma4:12b as the model for a sandboxed task. Codex is the OpenAI agent CLI. Useful when you already live in codex exec but want offline weights.

Crush + local OpenAI-compatible API

Point an OpenAI-compatible provider at Ollama or LiteRT-LM in crush.json (see Crush README configuration section). Same pattern as Open WebUI + local server setups.

Coding fit table (Gemma 4 12B itself)

API and cloud routes (gemma 4 12b api)

See our OpenRouter Free Models (2026) guide for router patterns; swap model IDs when 12B appears.

Troubleshooting

Who should use, watch, or skip

Related reading on AI Tools Radar

• Latest AI Models Compared (2026). Where Gemma 4 sits next to GPT-5.5 and DeepSeek V4
• DeepSeek V4 vs ChatGPT vs Claude for Coding (2026). If coding is your main stack
• Devin Desktop vs Cursor (2026). Picking an IDE agent lane beside local Gemma
• OpenRouter Free Models (2026). Hosted Gemma 4 31B patterns until 12B routes ship

Changelog

• 2026-06-05: First publish. Local setup for Ollama, MLX, LiteRT-LM, and 2026 coding agents (OpenCode, Codex CLI, Crush). Consumer Gemini CLI moves to Antigravity CLI.
• 2026-06-08 (update): Added Gemma 4 QAT (Quantization-Aware Training) checkpoints — Q4_0 GGUF, vLLM compressed tensors, and Unsloth QAT GGUFs for 12B. MTP QAT checkpoints also available.