Krasis documentation

Memory and speed

Understand the settings that affect model fit, response time and answer quality.

RAM and VRAM do different jobs

System RAM holds model backing data. GPU VRAM holds active weights, conversation state and temporary work buffers. Krasis moves routed experts to the GPU as needed, so a model can be larger than VRAM. It still needs enough RAM and transfer bandwidth.

Close other GPU-heavy programs before launching. Startup calibrates available VRAM, so the same model can receive a different expert-cache budget when other applications are using the GPU.

Context uses memory too

Context is the conversation the model can see: system instructions, messages, tool definitions, tool results and space for new output. Longer context needs more state and working memory.

Choose the context you actually need. Increasing the context limit can reduce the space available for resident experts, and a model’s advertised maximum does not mean it fits on your machine. A context cap and a cache-memory budget are separate controls; both must support the request.

Quantisation trades precision for space

Quantisation stores weights or conversation state with fewer bits. It reduces memory use, but can change answers. The launcher exposes separate controls for expert weights, attention weights and the KV cache (the attention state retained for earlier tokens).

Start with the launcher’s suggestions. HQQ6 attention and the Quality k6v6 cache are common launcher choices, but architecture-specific choices differ. DeepSeek-V4 uses Native or BF16 shared sequence state instead of conventional separate K/V formats.

Use a tested profile when quality matters. Increasing precision needs more memory; decreasing it needs a fresh quality check. BF16-heavy profiles are for validation, rather than the normal production setup.

The expert cache helps generation

HCS (Hot Cache Strategy) keeps frequently used experts in GPU memory. More useful resident experts can reduce transfers during generation. Dynamic HCS adjusts part of that residency as work changes.

The host-cache source mode saves system RAM by avoiding a duplicate mirror. mirror uses extra RAM for faster reloads. Leave this alone for a first run; the configuration reference explains the controls.

Conversation-prefix caching is different: it saves compatible conversation state so repeated prefixes may avoid work. Enable it deliberately and budget its RAM. It does not make every request cacheable.

Read speed results carefully

Prefill processes the input prompt. Decode generates new tokens. Time to the first token includes prompt processing; a long prompt can take time even when generation is fast.

Compare the same model, precision, prompt length and hardware. PCIe transfer speed matters when experts must move frequently. Internal engine timing and HTTP timing measure different things; use the labels on the benchmark page.

Change one setting at a time and check both speed and output quality. Leave per-layer timing instrumentation off when measuring normal performance.