Gemma 4 is here!

June 30, 2026

node-llama-cpp v3.19 is here, with full support for Gemma 4 models and improved performance!

---

Gemma 4

Gemma 4 comes in five flavors:

• gemma-4-E2B-it - 5.1B parameters with 2.3B active parameters, 128K context window
• gemma-4-E4B-it - 8B parameters with 4.5B active parameters, 128K context window
• gemma-4-12B-it - 11.95B parameters dense model, 256K context window
• gemma-4-26B-A4B-it - 25.2B parameters with 3.8B active parameters, 256K context window
• gemma-4-31B-it - 30.7B parameters dense model, 256K context window

Here are a few highlights of these models:

• The non-dense models are pretty fast due to the low number of effective parameters
• These are reasoning models, and you can disable reasoning when you don't need it
• They are very good at function calling and coding, and are built with agentic capabilities in mind
• The models family uses Sliding Window Attention (SWA), making 128K and 256K context windows much more efficient and thus practical
• They are provided with an Apache 2.0 license, so you can use them in your commercial applications

Recommended Models

Here are some recommended model URIs you can use to try out Gemma 4 right away:

Model	Size	Recommended URIs
Gemma 4 5B E2B	5.0GB	hf:giladgd/gemma-4-E2B-it-GGUF:Q8_0
Gemma 4 5B E2B	3.9GB	hf:giladgd/gemma-4-E2B-it-GGUF:Q6_K
Gemma 4 8B E4B	8.1GB	hf:giladgd/gemma-4-E4B-it-GGUF:Q8_0
Gemma 4 8B E4B	6.3GB	hf:giladgd/gemma-4-E4B-it-GGUF:Q6_K
Gemma 4 8B E4B	5.4GB	hf:giladgd/gemma-4-E4B-it-GGUF:Q4_K_M
Gemma 4 12B	12.7GB	hf:giladgd/gemma-4-12B-it-GGUF:Q8_0
Gemma 4 12B	9.8GB	hf:giladgd/gemma-4-12B-it-GGUF:Q6_K
Gemma 4 12B	8.5GB	hf:giladgd/gemma-4-12B-it-GGUF:Q5_K_M
Gemma 4 12B	7.4GB	hf:giladgd/gemma-4-12B-it-GGUF:Q4_K_M
Gemma 4 26B A4B MoE	26.9GB	hf:giladgd/gemma-4-26B-A4B-it-GGUF:Q8_0
Gemma 4 26B A4B MoE	22.6GB	hf:giladgd/gemma-4-26B-A4B-it-GGUF:Q6_K
Gemma 4 26B A4B MoE	19.1GB	hf:giladgd/gemma-4-26B-A4B-it-GGUF:Q5_K_M
Gemma 4 26B A4B MoE	16.8GB	hf:giladgd/gemma-4-26B-A4B-it-GGUF:Q4_K_M
Gemma 4 31B	32.6GB	hf:giladgd/gemma-4-31B-it-GGUF:Q8_0
Gemma 4 31B	25.2GB	hf:giladgd/gemma-4-31B-it-GGUF:Q6_K
Gemma 4 31B	21.8GB	hf:giladgd/gemma-4-31B-it-GGUF:Q5_K_M
Gemma 4 31B	18.7GB	hf:giladgd/gemma-4-31B-it-GGUF:Q4_K_M

TIP

Estimate the compatibility of a model with your machine before downloading it:

npx -y node-llama-cpp inspect estimate <model URI>

Try It Using the CLI

To quickly try out Gemma 4 5B E2B, you can use the CLI chat command:

npx -y node-llama-cpp chat --prompt "Hi there" hf:giladgd/gemma-4-E2B-it-GGUF:Q8_0

Long Context

One of the nicest things about Gemma 4 is how practical its long context is.

These models use hybrid attention: a mix of regular full-context attention and Sliding Window Attention (SWA). This allows the model to still make use of the broader context, while SWA keeps long contexts much cheaper to work with by focusing part of the attention to the recent window of tokens.

This makes long context much more practical, but reverting to a point in the context sequence before the SWA window can require reevaluating the tokens up to that point (unless you enable swaFullCache).

node-llama-cpp automatically takes lightweight checkpoints at strategic points in the sequence, so you don't have to worry about this. You can also manually take a checkpoint at strategic points in your workflow if you have a special case where you need more control.

Configuring Reasoning

Reasoning is enabled by default, but you can disable it or limit its budget if you want faster answers or to keep the token usage under control.

You can do so by either configuring Gemma4ChatWrapper, or by setting a reasoning budget for individual prompts.

import {
    getLlama, resolveModelFile, LlamaChatSession,
    Gemma4ChatWrapper
} from "node-llama-cpp";

const modelUri = "hf:giladgd/gemma-4-E2B-it-GGUF:Q8_0";


const llama = await getLlama();
const model = await llama.loadModel({
    modelPath: await resolveModelFile(modelUri)
});
const context = await model.createContext();
const session = new LlamaChatSession({
    contextSequence: context.getSequence(),
    chatWrapper: new Gemma4ChatWrapper({
        // Tell the model to not think:
        // reasoning: false
    })
});

const q1 = "Where do llamas come from?";
console.log("User: " + q1);

process.stdout.write("AI: ");
const a1 = await session.prompt(q1, {
    budgets: {
        // Force stop the model's reasoning when
        // it reaches 4096 tokens spent on thought segments:
        // thoughtTokens: 4096
    },
    onResponseChunk(chunk) {
        const isThoughtSegment = chunk.type === "segment" &&
            chunk.segmentType === "thought";
        const isCommentSegment = chunk.type === "segment" &&
            chunk.segmentType === "comment";
        
        if (chunk.type === "segment" && chunk.segmentStartTime != null)
            process.stdout.write(` [segment start: ${chunk.segmentType}] `);

        process.stdout.write(chunk.text);

        if (chunk.type === "segment" && chunk.segmentEndTime != null)
            process.stdout.write(` [segment end: ${chunk.segmentType}] `);
    }
});
process.stdout.write("\n\n");
console.log("AI: " + a1);

Stability and Performance Improvements

This release includes a myriad of stability and performance improvements across the board, but here are a few highlights:

• Flash attention is now enabled by default when supported
• mmap is automatically disabled when it could lead to better performance
• Grammar and function calling inference performance is now much faster
• Resource usage estimations are now much more accurate, so "auto" (the default) model and context configurations adapt to your machine much better
• Prebuilt binaries for arm architectures are now more optimized and should perform better
• Loading huge models won't crash the process anymore
• If in the past your machine felt laggy when loading a model to memory, it now should be a rare occurrence
• If in the past some models failed to load due to insufficient memory errors even though the machine had enough memory, then this issue should now be generally resolved

Resource Capping

In this release, you can set resource usage caps for a Llama instance to ensure that the total memory of all objects created from it won't exceed a certain amount of RAM or VRAM. This should help you ensure that node-llama-cpp plays nicely with other GPU workloads that you might run from the same process.

NOTE

Setting the caps too low may prevent you from loading models at all, so use with caution.

When using caps, make sure to provide users with the ability to configure the caps themselves to avoid bad defaults, since there's no one-size-fits-all configuration for all machines.

import {
    getLlama, resolveModelFile
} from "node-llama-cpp";

const modelUri = "hf:giladgd/gemma-4-E2B-it-GGUF:Q8_0";


const llama = await getLlama();
llama.setVramCap(4 * Math.pow(1024, 3)); // 4GB VRAM cap

const model = await llama.loadModel({
    modelPath: await resolveModelFile(modelUri)
});
const context = await model.createContext();
// the total of the model and context VRAM won't exceed 6GB

Behind the Scenes

The making of this release took longer than usual, as it includes many infrastructural improvements that required a more thorough testing to ensure everything works as expected across a wide range of machines and configurations.

While working on this release, I've got a new machine to test on, and received help from community members who tested the PR and spotted issues on their machines.

These improvements mean that in the future it'll take much less effort to support new models as they come out, and that node-llama-cpp will perform even better across a wider range of machines and configurations.

If your project or company relies on node-llama-cpp, consider becoming a sponsor to accelerate the development of new features, improve hardware coverage, and expand testing.

Until the next release,
♥️ node-llama-cpp maintainers.