KRASISGitHub

Krasis documentation

Configuration reference

Save a working profile, override settings and understand the main server controls.

On this page

Start from a saved profile

Use the launcher or Manager to create a working profile first. Launcher profiles live under ~/.krasis/config (or %USERPROFILE%\.krasis\config on Windows). Reusing a generated profile preserves model-specific settings that a minimal hand-written file may miss.

Config files use KEY=VALUE lines. Start one with:

krasis --config /path/to/profile.conf

Command-line flags override values in that config. For example, to change only the port:

krasis --config /path/to/profile.conf --port 8013

The tables below cover common controls. For the complete catalogue and specialised modes, use the 1.0.22 advanced reference.

Server and GPU selection

Flag Purpose
--model-path PATH Local Hugging Face checkpoint directory with safetensors and config files.
--selected-gpus 0,2 Select visible GPU indices; verify the inventory on the machine running the server.
--num-gpus N Limit how many GPUs to use.
--multi-gpu-mode auto Measure and choose a topology supported by the loaded execution graph.
--pp-partition 24,24 Explicit layer partition for a compatible multi-GPU setup.
--host ADDRESS Bind address; default 0.0.0.0. Use 127.0.0.1 for local-only access.
--port N Inference server port; default 8012.
--api-keys-file PATH Custom authentication file; see API keys.

A model’s architecture determines which multi-GPU topologies are available. Selecting auto does not imply that every architecture supports expert-serving peers.

Common saved keys

These are examples of individual lines to edit in an existing generated profile, not a complete model configuration:

CFG_HOST="127.0.0.1"
CFG_PORT="8012"
CFG_MAX_CONTEXT_TOKENS="8192"
CFG_API_KEYS_FILE="/absolute/path/to/keys.json"

An empty or unset CFG_API_KEYS_FILE selects the default key-file location. A custom file must already exist and be valid at startup. Relative paths resolve from the server’s working directory.

Precision controls

Flag Choices and meaning
--gpu-expert-bits 4 or 8; default 4.
--cpu-expert-bits 4 or 8; default 4.
--attention-quant HQQ attention modes or bf16. Launcher suggestions differ from direct-server defaults.
--kv-dtype Conventional k6v6, k4v4 or bf16 where supported; architecture-specific state has its own choices.
--shared-expert-quant int8 or bf16; default int8.
--dense-mlp-quant int8 or bf16; default int8.
--lm-head-quant int8 or bf16; default int8.
--vision-quant int4 or validation bf16; vision capability must be assessed separately.

The launcher offers HQQ4 and HQQ6 plus mixed promotion presets. Use its generated profile to preserve the exact representation of a mixed preset. Direct-server attention defaults to BF16, while fresh launcher selections normally use HQQ6. Old profiles can retain older choices.

AWQ attention and Polar4 cache are deprecated and disabled for new runs. BF16-heavy modes are validation profiles. See Memory and speed before changing precision.

Context and memory

Flag Meaning
--max-context-tokens N Cap the effective context; 0 uses the model-declared limit. Actual memory must still support it.
--kv-cache-mb N Cache-memory budget; documented default 1000 MB.
--vram-safety-margin N Reserved VRAM safety margin in MB; documented default 600.
--stream-attention Stream attention weights from CPU for large models.
--hcs-host-cache-mode source RAM-saving host backing; mirror duplicates data for faster reloads, auto chooses according to available RAM.
--dynamic-hcs-tail-blocks N Recency-adaptive expert-cache tail; valid 1..5, default 2.
--approved-heatmap-mode MODE auto uses a compatible approved heatmap with local fallback; off uses local startup capture; require fails without a compatible approved artifact.

The corresponding saved keys include CFG_MAX_CONTEXT_TOKENS, CFG_KV_CACHE_MB, CFG_KV_DTYPE, CFG_HCS and CFG_HCS_HOST_CACHE_MODE. HCS expert residency is separate from conversation-prefix caching.

Conversation-prefix caching

To enable RAM-backed prefix-state caching in a saved profile:

CFG_PREFIX_CACHE="1"
CFG_PREFIX_CACHE_RAM_FRACTION="0.25"

The fraction must be greater than zero and at most one. The budget follows available RAM rather than reserving a fixed amount. Cache hits require an exact compatible prefix and runtime signature. Image-input, speculative-decode and multi-GPU continuation requests have additional unsupported modes in this release.

Fresh launcher selections and the direct server parser enable this feature by default. Existing saved profiles can override it; set it explicitly when reproducibility matters. Statistics are available at GET /v1/session-cache/stats, with an API key when authentication is enabled.

Remote access and diagnostics

--ssh-tunnel user@host sets up a reverse SSH tunnel; an optional :ssh_port selects the SSH port. --ssh-key-path PATH selects a key. The remote loopback port forwards to the local inference server. Use an existing authorised SSH destination.

--benchmark-only runs a benchmark and exits. --timing adds per-layer instrumentation and overhead; leave it off for normal speed measurements. Experimental pruning, CPU-tail and speculative settings need their own quality and compatibility checks. See the full reference before enabling them.