Run Gemma 4 Locally with Ollama

You want Gemma 4 to read private code or documents without sending them to a hosted API. Ollama binds to 127.0.0.1:11434 by default, and that local HTTP API doesn't require authentication, so don't expose the port without a real network and access plan.^{[1]Reference 1FAQ - Ollamahttps://docs.ollama.com/faq[2]Reference 2Authenticationhttps://docs.ollama.com/api/authentication}

Gemma 4 is Google DeepMind's Apache 2.0 open-weights family: E2B, E4B, 12B, 26B A4B, and 31B. The "E" in E2B and E4B means effective parameters. Those two edge models use per-layer embeddings, so the stored embedding tables are larger than the effective count. E2B and E4B have 128K model context limits. 12B, 26B A4B, and 31B have 256K limits.^{[3]Reference 3Gemma 4 Model Cardhttps://ai.google.dev/gemma/docs/core/model_card_4} Those numbers are architecture ceilings, not a promise that your Ollama process can allocate them.

Google added the 12B unified model on June 3, 2026 and positions it for machines with 16 GB of VRAM or unified memory. Start with gemma4:12b when its 7.6 GB artifact still leaves room for context and runtime overhead. Fall back to gemma4:e4b-it-qat when memory is tighter. Ollama's published local tags currently accept text and images. Google's model card also lists native audio for E2B, E4B, and 12B, so treat local audio as unsupported until the exact Ollama tag says otherwise.^{[4]Reference 4Introducing Gemma 4 12B: a unified, encoder-free multimodal modelhttps://blog.google/innovation-and-ai/technology/developers-tools/introducing-gemma-4-12b/[5]Reference 5gemma4https://ollama.com/library/gemma4[6]Reference 6gemma4 tagshttps://ollama.com/library/gemma4/tags}

Choose an exact Ollama tag

An Ollama tag names the variant and the packaged artifact. It can also select quantization or a runtime path. As of August 13, 2026, these are useful starting points:^{[6]Reference 6gemma4 tagshttps://ollama.com/library/gemma4/tags}

⚠️ Cloud tags: gemma4:cloud and gemma4:31b-cloud call Ollama cloud models. They aren't local artifacts, so don't use them for an offline or local-data boundary.^{[6]Reference 6gemma4 tagshttps://ollama.com/library/gemma4/tags[1]Reference 1FAQ - Ollamahttps://docs.ollama.com/faq}

The 26B A4B model is a Mixture-of-Experts model with 25.2B total parameters and 3.8B active parameters per token. Ollama's library page rounds the active path to 4B. All expert weights still need to load for fast routing, so the memory footprint tracks the ~26B total, not a dense 4B model.^{[3]Reference 3Gemma 4 Model Cardhttps://ai.google.dev/gemma/docs/core/model_card_4[5]Reference 5gemma4https://ollama.com/library/gemma4}

💡 Fit rule: Keep the smallest exact tag that passes your task evaluation at the context length you actually use.

Don't pin gemma4:latest in scripts. Ollama currently maps that alias to E4B (9.6 GB), and Ollama can retarget latest without your scripts noticing. An exact tag makes later benchmarks and rollback easier.^{[5]Reference 5gemma4https://ollama.com/library/gemma4}

Budget runtime memory

Published artifact size isn't the full runtime requirement. Add memory for metadata, runtime buffers, prompt processing, the key-value (KV) cache, and a safety margin. The KV cache stores attention state from earlier tokens, so longer context and concurrent requests raise memory use. Inference: TTFT, TPS & KV Cache is the place to size that cache on purpose instead of guessing from download size.

On Google's 16 GB laptop target for 12B, leftover after the published Ollama artifact is the first filter. gemma4:12b leaves 8.4 GB. gemma4:e4b-it-qat leaves 9.9 GB. gemma4:26b-a4b-it-qat leaves none, so KV cache and the OS share that 16 GB with the weights.^{[4]Reference 4Introducing Gemma 4 12B: a unified, encoder-free multimodal modelhttps://blog.google/innovation-and-ai/technology/developers-tools/introducing-gemma-4-12b/[6]Reference 6gemma4 tagshttps://ollama.com/library/gemma4/tags}

Hardware capacity alone doesn't determine speed. On a discrete GPU, partial CPU offload can hurt latency. On Apple Silicon, CPU and GPU share unified memory, so read ollama ps placement labels next to measured throughput. Benchmark the exact machine, tag, context, and workload. Local LLM Deployment walks through the full weights-plus-cache-plus-reserve budget, including when to roll back a tag.

What QAT and MTP change

Quantization-aware training (QAT) and multi-token prediction (MTP) solve different problems.

QAT simulates low-precision behavior during training so the model can adapt before weights are compressed. Google's official QAT release covers Q4_0 checkpoints and mobile-specific formats. Ollama's default aliases and QAT tags use different quantization formats, so the published size gap mixes QAT with packaging. That gap is large for E2B (7.2 GB to 4.3 GB) and E4B (9.6 GB to 6.1 GB), and small for 12B (7.6 GB to 7.2 GB). Treat each tag as its own artifact instead of assuming one percentage.^{[7]Reference 7Gemma 4 QAT models: Optimizing model compression for mobile and laptop efficiencyhttps://blog.google/innovation-and-ai/technology/developers-tools/quantization-aware-training-gemma-4/[6]Reference 6gemma4 tagshttps://ollama.com/library/gemma4/tags} Model Quantization: GPTQ, AWQ & GGUF explains how QAT differs from post-training methods and why GGUF is a container rather than a bit width.

MTP is a speculative-decoding path. A small drafter proposes several tokens, the target model verifies the proposals in parallel, then the runtime accepts the matching prefix and corrects the first rejected position. It can raise generated-token throughput. It doesn't shrink target weights, and it doesn't remove prompt-processing or KV-cache cost.^{[8]Reference 8Accelerating Gemma 4: faster inference with multi-token prediction draftershttps://blog.google/innovation-and-ai/technology/developers-tools/multi-token-prediction-gemma-4/}

Ollama's documented MTP path is runtime-specific. Ollama 0.31 added automatic MTP for Gemma 4 on Apple Silicon MLX. The published recipe is to update Ollama and re-pull gemma4:12b-mlx. No MTP flag is required. Ollama reports nearly 90% faster generation on Gemma 4 12B (nvfp4) on an M5 Max running the Aider polyglot workload, and says the gain depends on workload, hardware, quantization, and draft acceptance. Treat that as a source-specific result, not expected speed on every machine.^{[9]Reference 9Faster Gemma 4 on MLX with Multi-Token Predictionhttps://ollama.com/blog/faster-gemma-4-mlx-mtp}

The separate gemma4:31b-coding-mtp-bf16 tag is 64 GB and isn't a laptop default. For non-MLX tags, don't assume MTP turns on just because Google publishes drafter weights. Check the Ollama release notes and the exact tag, then compare measured eval_duration with the same prompt and sampling.^{[6]Reference 6gemma4 tagshttps://ollama.com/library/gemma4/tags} Speculative Decoding derives the draft-and-verify loop MTP is using.

Install from the official package

Once the tag is pinned, install Ollama from the official package and pull only that tag. On macOS and Windows, use the desktop installer from the official Ollama download page. The official Linux guide offers an install script. Download that script first so you can inspect the file before you run it, or use the documented manual package path when the environment needs stricter supply-chain controls.^{[10]Reference 10Ollama GitHub Repositoryhttps://github.com/ollama/ollama[1]Reference 1FAQ - Ollamahttps://docs.ollama.com/faq}

Pull one exact local tag. Check free disk first, because the pull stores several gigabytes.

ollama ps shows the loaded model, allocated context, and PROCESSOR placement. A generated reply proves the basic path works. It doesn't prove latency, memory margin, or task quality.^{[1]Reference 1FAQ - Ollamahttps://docs.ollama.com/faq}

If a local HTTP check fails, separate a down server from a missing model. These checks use jq:

On a Linux manual install, start ollama serve in another terminal if no service is running. Desktop apps normally manage the server. Keep the bind address at 127.0.0.1 unless the deployment has a deliberate authentication, firewall, and TLS plan.^{[1]Reference 1FAQ - Ollamahttps://docs.ollama.com/faq[2]Reference 2Authenticationhttps://docs.ollama.com/api/authentication}

Call the native API

The native /api/generate endpoint exposes runtime timing fields and accepts num_ctx, which makes it the best first benchmark surface. Disable streaming so one JSON object contains the text and the timing data:^{[11]Reference 11Generate a Responsehttps://docs.ollama.com/api/generate[12]Reference 12Usagehttps://docs.ollama.com/api/usage}

Ollama reports durations in nanoseconds. Generated-token rate is eval_count / (eval_duration / 1e9). Prompt rate uses prompt_eval_count and prompt_eval_duration the same way.^{[12]Reference 12Usagehttps://docs.ollama.com/api/usage}

This jq expression prints both rates and guards against missing or zero values:

In Python, a benchmark evaluator parses the response receipt, calculates prompt and decode throughputs, and verifies whether the run meets a local latency target:

Ollama can preload a model with an empty generate request, which is a clean warm-up before you record rates:^{[1]Reference 1FAQ - Ollamahttps://docs.ollama.com/faq}

🎯 Measurement rule: Warm the model once, then repeat the exact prompt with tag, context, sampling, output cap, Ollama version, and machine load held constant.

Use the OpenAI-compatible API carefully

Ollama supports parts of /v1/chat/completions and non-stateful /v1/responses. Point the client at http://127.0.0.1:11434/v1/ and use a placeholder key such as ollama. Check supported fields before you move a hosted integration, because this API isn't a full drop-in replacement.^{[13]Reference 13OpenAI compatibility - Ollamahttps://docs.ollama.com/api/openai-compatibility}

OpenAI-compatible requests don't set Ollama context size per request. To pin context, create a derived model:

Then create it and call the derived name:

Ollama's OpenAI compatibility docs say /v1/responses arrived in Ollama 0.13.3 and doesn't support previous_response_id or conversation state. Keep conversation state in the application if the integration depends on Responses API semantics.^{[13]Reference 13OpenAI compatibility - Ollamahttps://docs.ollama.com/api/openai-compatibility}

Set context you can afford

Ollama currently chooses default allocated context by available VRAM: below 24 GiB gets 4K, 24 to 48 GiB gets 32K, and 48 GiB or more gets 256K. Its FAQ still gives 4096 as a general baseline, so inspect the running allocation instead of trusting either number for the installed version.^{[14]Reference 14Context length - Ollamahttps://docs.ollama.com/context-length[1]Reference 1FAQ - Ollamahttps://docs.ollama.com/faq}

Ollama's context docs prefer the model's maximum when the weights stay on GPU. On a laptop that often isn't possible, so set the smallest context that covers the task. Through the native API, use "options": {"num_ctx": 8192}. In the interactive CLI, use /set parameter num_ctx 8192. For a standalone server process, set OLLAMA_CONTEXT_LENGTH=8192 ollama serve.

After you change it, run ollama ps and check CONTEXT plus PROCESSOR. If the model spills or the machine swaps, shorten context or pick a smaller QAT tag. The 128K or 256K model limit isn't a sensible default on a small machine.

Keep the template and modalities inside the runtime

Gemma 4 uses standard system, user, and assistant roles. Google recommends sampling with temperature 1.0, top-p 0.95, and top-k 64 for general model behavior. For the deterministic benchmark above, temperature 0 is a test control, not Google's quality recommendation.^{[3]Reference 3Gemma 4 Model Cardhttps://ai.google.dev/gemma/docs/core/model_card_4[5]Reference 5gemma4https://ollama.com/library/gemma4}

Thinking turns on when the system prompt starts with <|think|>. In multi-turn history, Google says keep the final answer and omit prior thoughts, except on tool-call turns where thinking content should stay. Ollama applies the model chat template when you use chat endpoints, so pass roles and content instead of assembling control tokens by hand.^{[3]Reference 3Gemma 4 Model Cardhttps://ai.google.dev/gemma/docs/core/model_card_4[5]Reference 5gemma4https://ollama.com/library/gemma4}

Google's model card says to put image content before the text prompt and audio content after it. Ollama's Gemma 4 page currently says to put image and audio before the text. For the local tags that advertise text and image only, put the image first and let Ollama apply the template. Don't build a local audio workflow from Google's model-level audio support.^{[3]Reference 3Gemma 4 Model Cardhttps://ai.google.dev/gemma/docs/core/model_card_4[5]Reference 5gemma4https://ollama.com/library/gemma4[6]Reference 6gemma4 tagshttps://ollama.com/library/gemma4/tags}

Diagnose from the symptom

Don't change tag, prompt, context, sampling, and runtime together. One changed variable lets the run receipt answer one question.

Build a reproducible local comparison

Use three prompt classes: short structured extraction, a realistic repository explanation, and fixed-length generation. Warm each tag once, run several measured trials, and store the raw JSON. Record:

Latency isn't quality. Write task checks before you compare, such as valid JSON schema, correct failing assertions, or a cited answer from a supplied file. Keep the smallest tag that meets the quality and latency budget under real context.