mirror of
https://github.com/LostRuins/koboldcpp.git
synced 2026-08-10 17:06:13 +00:00
4ae84dea27
* server: add an ssh transport to the tools runtime --tools-runtime ssh:<target> runs the built-in tools on a remote host, where target is whatever ssh already resolves, a user@host or a config alias, so no credentials live in llama.cpp. Only build_argv and upload differ from the docker transport: the remote shell re-parses the command line, so the argv travels through shell_quote_join, and files go over scp with the same quoting on the remote path. Authentication is key-based and the host key must already be trusted, since the tools run without a console and any prompt would hang them. The target is validated before use. The spec can reach us from the x-tool-runtime header, and a leading dash would turn it into an ssh option, which is enough to run a command back on the host. Nothing is created and nothing is reclaimed, so an ssh spec goes straight to the tool call instead of through the container runtime. Note that this is remoting rather than isolation: the tools can do whatever the target account can do, and the isolation is whatever runs them on the far side. * server: support podman in the tools runtime docker and podman expose the same run, exec, cp and inspect verbs with the same argument order, so a single implementation drives both and the engine is carried by the spec prefix: podman:<image> and podman-container:<id> sit next to the docker forms. tools_io_docker becomes tools_io_container and the runtime spawner becomes server_tools_container_runtime, both holding the client binary chosen at parse time. A single parse_container_runtime() resolves every spec, so adding another engine is one string in the table. make_tools_io() now rejects the spawning forms. The spec also reaches it from the x-tool-runtime header, which is client controlled, and only the runtime that owns a container is allowed to create one: a tool call can attach to a running container, nothing more. * ./build/bin/llama-gen-docs * server: simplify the tools runtime and drop the file copy step A server_tools_runtime base with one virtual spec() replaces the container runtime and the bare spec string that ssh needed next to it, so server_tools is back to a single pointer and neither setup nor the handler tests which of the two is set. write_file used to spill its content into a temporary file on the host and copy it in, because run_subprocess had no way to feed a child. It now takes an optional stdin payload and creates the parent directory and the file in a single round trip through a shell in the isolate. That removes the upload virtual and both implementations: no more container cp or scp, no second binary on the host, no sftp subsystem on the target, no predictable temporary in a shared tmp, and none of the content reaching an argv the remote shell re-parses. It also fixes write_file over ssh, which never worked: scp speaks sftp and takes the remote path literally, so quoting it kept the quotes in the file name. Writing the payload before reading the output relies on the child draining stdin as it goes, which holds for cat, its only user today. * ./build/bin/llama-gen-docs * server: harden the tools runtime against argv injection and a stdin stall Validate the container id from x-tool-runtime and --tools-runtime the same way the ssh target already is, so an id shaped like an option (docker-container:--privileged) is rejected before it reaches the engine's exec command line instead of running against a hardened container. Feed the child's stdin after the watchdog is armed, so a transport that stalls mid-write is terminated at the deadline rather than blocking the request forever. Cover both guards and fix the unknown-scheme test, which used ssh: as its example and now names a real runtime. * tests: exercise the tools runtime tests on podman as well as docker Follow-up #26507. The container runtime drives docker and podman through one implementation, so parametrize the availability helper, the container fixture and the attach test on the engine, and cover both engine prefixes in the container id injection test. Each engine skips on its own when it is not installed. The spawn cleanup test stays docker only: it recovers the spawned id from the container hostname, which docker sets to the short id and podman rootless does not guarantee. Podman keeps its coverage through the attach path. * server: release the container handle before respawning Follow-up #26507. create() writes over the handle it is given, so a respawn after the container died on its own leaked the pipes and the process handle of the previous one. * server: trim the tools runtime comments * server: read tool output as raw bytes and harden the runtime on Windows The stdout pipe is read with read() instead of fgets(), so a chunk can hold any byte, including NUL, and still streams as soon as data is available. Past the size cap the pipe keeps draining so the child never blocks on a full pipe. Both pipe fds are forced to binary mode on Windows, where the CRT defaults them to text mode and translates line endings in both directions. Stdin is now always closed after the feed: the child reads a deterministic EOF, and the Windows docker and ssh clients stop outliving their command on a stdin pipe that never closes. The attach form of --tools-runtime has no lifecycle to own, so it becomes a static target validated once at startup. This removes the subprocess that ran on every tool call and serialized calls behind a mutex; a stopped container now surfaces the engine's own error at exec time. The cidfile path is passed as UTF-8, matching the encoding the subprocess layer expects for the CreateProcessW command line, so the spawn form works from a non-ASCII Windows profile. The SIGPIPE note in server.cpp now names the tools runtime children as well as the MCP ones. * clean up comments * less pollute global scope * nits * tests: name the container image after both engines --------- Co-authored-by: Xuan Son Nguyen <son@huggingface.co> |
||
|---|---|---|
| .. | ||
| cli-client.cpp | ||
| cli-client.h | ||
| cli-context.cpp | ||
| cli-context.h | ||
| cli-server.h | ||
| cli-ui.h | ||
| cli.cpp | ||
| CMakeLists.txt | ||
| main.cpp | ||
| README.md | ||
llama.cpp/tools/cli
Usage
Common params
| Argument | Explanation |
|---|---|
-h, --help, --usage |
print usage and exit |
--version |
show version and build info |
-cl, --cache-list |
show list of models in cache |
--completion-bash |
print source-able bash completion script for llama.cpp |
-t, --threads N |
number of CPU threads to use during generation (default: -1) (env: LLAMA_ARG_THREADS) |
-tb, --threads-batch N |
number of threads to use during batch and prompt processing (default: same as --threads) |
-C, --cpu-mask M |
CPU affinity mask: arbitrarily long hex. Complements cpu-range (default: "") |
-Cr, --cpu-range lo-hi |
range of CPUs for affinity. Complements --cpu-mask |
--cpu-strict <0|1> |
use strict CPU placement (default: 0) |
--prio N |
set process/thread priority : low(-1), normal(0), medium(1), high(2), realtime(3) (default: 0) |
--poll <0...100> |
use polling level to wait for work (0 - no polling, default: 50) |
-Cb, --cpu-mask-batch M |
CPU affinity mask: arbitrarily long hex. Complements cpu-range-batch (default: same as --cpu-mask) |
-Crb, --cpu-range-batch lo-hi |
ranges of CPUs for affinity. Complements --cpu-mask-batch |
--cpu-strict-batch <0|1> |
use strict CPU placement (default: same as --cpu-strict) |
--prio-batch N |
set process/thread priority : 0-normal, 1-medium, 2-high, 3-realtime (default: 0) |
--poll-batch <0|1> |
use polling to wait for work (default: same as --poll) |
-c, --ctx-size N |
size of the prompt context (default: 0, 0 = loaded from model) (env: LLAMA_ARG_CTX_SIZE) |
-n, --predict, --n-predict N |
number of tokens to predict (default: -1, -1 = infinity) (env: LLAMA_ARG_N_PREDICT) |
-b, --batch-size N |
logical maximum batch size (default: 2048) (env: LLAMA_ARG_BATCH) |
-ub, --ubatch-size N |
physical maximum batch size (default: 512) (env: LLAMA_ARG_UBATCH) |
--keep N |
number of tokens to keep from the initial prompt (default: 0, -1 = all) |
--swa-full |
use full-size SWA cache (default: false) (more info) (env: LLAMA_ARG_SWA_FULL) |
-fa, --flash-attn [on|off|auto] |
set Flash Attention use ('on', 'off', or 'auto', default: 'auto') (env: LLAMA_ARG_FLASH_ATTN) |
-p, --prompt PROMPT |
prompt to start generation with; for system message, use -sys |
--perf, --no-perf |
whether to enable internal libllama performance timings (default: false) (env: LLAMA_ARG_PERF) |
-f, --file FNAME |
a file containing the prompt (default: none) |
-bf, --binary-file FNAME |
binary file containing the prompt (default: none) |
-e, --escape, --no-escape |
whether to process escapes sequences (\n, \r, \t, ', ", \) (default: true) |
--rope-scaling {none,linear,yarn} |
RoPE frequency scaling method, defaults to linear unless specified by the model (env: LLAMA_ARG_ROPE_SCALING_TYPE) |
--rope-scale N |
RoPE context scaling factor, expands context by a factor of N (env: LLAMA_ARG_ROPE_SCALE) |
--rope-freq-base N |
RoPE base frequency, used by NTK-aware scaling (default: loaded from model) (env: LLAMA_ARG_ROPE_FREQ_BASE) |
--rope-freq-scale N |
RoPE frequency scaling factor, expands context by a factor of 1/N (env: LLAMA_ARG_ROPE_FREQ_SCALE) |
--yarn-orig-ctx N |
YaRN: original context size of model (default: 0 = model training context size) (env: LLAMA_ARG_YARN_ORIG_CTX) |
--yarn-ext-factor N |
YaRN: extrapolation mix factor (default: -1.00, 0.0 = full interpolation) (env: LLAMA_ARG_YARN_EXT_FACTOR) |
--yarn-attn-factor N |
YaRN: scale sqrt(t) or attention magnitude (default: -1.00) (env: LLAMA_ARG_YARN_ATTN_FACTOR) |
--yarn-beta-slow N |
YaRN: high correction dim or alpha (default: -1.00) (env: LLAMA_ARG_YARN_BETA_SLOW) |
--yarn-beta-fast N |
YaRN: low correction dim or beta (default: -1.00) (env: LLAMA_ARG_YARN_BETA_FAST) |
-kvo, --kv-offload, -nkvo, --no-kv-offload |
whether to enable KV cache offloading (default: enabled) (env: LLAMA_ARG_KV_OFFLOAD) |
--repack, -nr, --no-repack |
whether to enable weight repacking (default: enabled) (env: LLAMA_ARG_REPACK) |
--no-host |
bypass host buffer allowing extra buffers to be used (env: LLAMA_ARG_NO_HOST) |
-ctk, --cache-type-k TYPE |
KV cache data type for K allowed values: f32, f16, bf16, q8_0, q4_0, q4_1, iq4_nl, q5_0, q5_1 (default: f16) (env: LLAMA_ARG_CACHE_TYPE_K) |
-ctv, --cache-type-v TYPE |
KV cache data type for V allowed values: f32, f16, bf16, q8_0, q4_0, q4_1, iq4_nl, q5_0, q5_1 (default: f16) (env: LLAMA_ARG_CACHE_TYPE_V) |
-dt, --defrag-thold N |
KV cache defragmentation threshold (DEPRECATED) (env: LLAMA_ARG_DEFRAG_THOLD) |
-np, --parallel N |
number of parallel sequences to decode (default: 1) (env: LLAMA_ARG_N_PARALLEL) |
--rpc SERVERS |
comma-separated list of RPC servers (host:port) (env: LLAMA_ARG_RPC) |
--mlock |
DEPRECATED in favor of --load-mode: force system to keep model in RAM rather than swapping or compressing(env: LLAMA_ARG_MLOCK) |
--mmap, --no-mmap |
DEPRECATED in favor of --load-mode: whether to memory-map model. (if mmap disabled, slower load but may reduce pageouts if not using mlock)(env: LLAMA_ARG_MMAP) |
-dio, --direct-io, -ndio, --no-direct-io |
DEPRECATED in favor of --load-mode: use DirectIO if available(env: LLAMA_ARG_DIO) |
-lm, --load-mode MODE |
model loading mode (default: mmap) - none: no special loading mode - mmap: memory-map model (if mmap disabled, slower load but may reduce pageouts if not using mlock) - mlock: force system to keep model in RAM rather than swapping or compressing - mmap+mlock: mmap + force system to keep model in RAM rather than swapping or compressing - dio: use DirectIO if available (env: LLAMA_ARG_LOAD_MODE) |
--numa TYPE |
attempt optimizations that help on some NUMA systems - distribute: spread execution evenly over all nodes - isolate: only spawn threads on CPUs on the node that execution started on - numactl: use the CPU map provided by numactl if run without this previously, it is recommended to drop the system page cache before using this see https://github.com/ggml-org/llama.cpp/issues/1437 (env: LLAMA_ARG_NUMA) |
-dev, --device <dev1,dev2,..> |
comma-separated list of devices to use for offloading (none = don't offload) use --list-devices to see a list of available devices (env: LLAMA_ARG_DEVICE) |
--list-devices |
print list of available devices and exit |
-ot, --override-tensor <tensor name pattern>=<buffer type>,... |
override tensor buffer type (env: LLAMA_ARG_OVERRIDE_TENSOR) |
-cmoe, --cpu-moe |
keep all Mixture of Experts (MoE) weights in the CPU (env: LLAMA_ARG_CPU_MOE) |
-ncmoe, --n-cpu-moe N |
keep the Mixture of Experts (MoE) weights of the first N layers in the CPU (env: LLAMA_ARG_N_CPU_MOE) |
-ngl, --gpu-layers, --n-gpu-layers N |
max. number of layers to store in VRAM, either an exact number, 'auto', or 'all' (default: auto) (env: LLAMA_ARG_N_GPU_LAYERS) |
-sm, --split-mode {none,layer,row,tensor} |
how to split the model across multiple GPUs, one of: - none: use one GPU only - layer (default): split layers and KV across GPUs (pipelined) - row: split weight across GPUs by rows (parallelized) - tensor: split weights and KV across GPUs (parallelized, EXPERIMENTAL) (env: LLAMA_ARG_SPLIT_MODE) |
-ts, --tensor-split N0,N1,N2,... |
fraction of the model to offload to each GPU, comma-separated list of proportions, e.g. 3,1 (env: LLAMA_ARG_TENSOR_SPLIT) |
-mg, --main-gpu INDEX |
the GPU to use for the model (with split-mode = none), or for intermediate results and KV (with split-mode = row) (default: 0) (env: LLAMA_ARG_MAIN_GPU) |
-fit, --fit [on|off] |
whether to adjust unset arguments to fit in device memory ('on' or 'off', default: 'on') (env: LLAMA_ARG_FIT) |
-fitt, --fit-target MiB0,MiB1,MiB2,... |
target margin per device for --fit, comma-separated list of values, single value is broadcast across all devices, default: 1024 (env: LLAMA_ARG_FIT_TARGET) |
-fitc, --fit-ctx N |
minimum ctx size that can be set by --fit option, default: 4096 (env: LLAMA_ARG_FIT_CTX) |
--check-tensors |
check model tensor data for invalid values (default: false) |
--override-kv KEY=TYPE:VALUE,... |
advanced option to override model metadata by key. to specify multiple overrides, either use comma-separated values. types: int, float, bool, str. example: --override-kv tokenizer.ggml.add_bos_token=bool:false,tokenizer.ggml.add_eos_token=bool:false |
--op-offload, --no-op-offload |
whether to offload host tensor operations to device (default: true) |
--lora FNAME |
path to LoRA adapter (use comma-separated values to load multiple adapters) |
--lora-scaled FNAME:SCALE,... |
path to LoRA adapter with user defined scaling (format: FNAME:SCALE,...) note: use comma-separated values |
--control-vector FNAME |
add a control vector note: use comma-separated values to add multiple control vectors |
--control-vector-scaled FNAME:SCALE,... |
add a control vector with user defined scaling SCALE note: use comma-separated values (format: FNAME:SCALE,...) |
--control-vector-layer-range START END |
layer range to apply the control vector(s) to, start and end inclusive |
-m, --model FNAME |
model path to load (env: LLAMA_ARG_MODEL) |
-mu, --model-url MODEL_URL |
model download url (default: unused) (env: LLAMA_ARG_MODEL_URL) |
-dr, --docker-repo [<repo>/]<model>[:quant] |
Docker Hub model repository. repo is optional, default to ai/. quant is optional, default to :latest. example: gemma3 (default: unused) (env: LLAMA_ARG_DOCKER_REPO) |
-hf, -hfr, --hf-repo <user>/<model>[:quant] |
Hugging Face model repository; quant is optional, case-insensitive, default to Q4_K_M, or falls back to the first file in the repo if Q4_K_M doesn't exist. mmproj is also downloaded automatically if available. to disable, add --no-mmproj example: ggml-org/GLM-4.7-Flash-GGUF:Q4_K_M (default: unused) (env: LLAMA_ARG_HF_REPO) |
-hff, --hf-file FILE |
Hugging Face model file. If specified, it will override the quant in --hf-repo (default: unused) (env: LLAMA_ARG_HF_FILE) |
-hft, --hf-token TOKEN |
Hugging Face access token (default: value from HF_TOKEN environment variable) (env: HF_TOKEN) |
--log-disable |
Log disable |
--log-file FNAME |
Log to file (env: LLAMA_ARG_LOG_FILE) |
--log-colors [on|off|auto] |
Set colored logging ('on', 'off', or 'auto', default: 'auto') 'auto' enables colors when output is to a terminal (env: LLAMA_ARG_LOG_COLORS) |
-v, --verbose, --log-verbose |
Set verbosity level to infinity (i.e. log all messages, useful for debugging) |
--offline |
Offline mode: forces use of cache, prevents network access (env: LLAMA_ARG_OFFLINE) |
-lv, --verbosity, --log-verbosity N |
Set the verbosity threshold. Messages with a higher verbosity will be ignored. Values: - 0: generic output - 1: error - 2: warning - 3: info - 4: trace (more info) - 5: debug (default: 3) (env: LLAMA_ARG_LOG_VERBOSITY) |
--log-prefix, --no-log-prefix |
Enable prefix in log messages (env: LLAMA_ARG_LOG_PREFIX) |
--log-timestamps, --no-log-timestamps |
Enable timestamps in log messages (env: LLAMA_ARG_LOG_TIMESTAMPS) |
--spec-draft-type-k, -ctkd, --cache-type-k-draft TYPE |
KV cache data type for K for the draft model allowed values: f32, f16, bf16, q8_0, q4_0, q4_1, iq4_nl, q5_0, q5_1 (default: f16) (env: LLAMA_ARG_SPEC_DRAFT_CACHE_TYPE_K) |
--spec-draft-type-v, -ctvd, --cache-type-v-draft TYPE |
KV cache data type for V for the draft model allowed values: f32, f16, bf16, q8_0, q4_0, q4_1, iq4_nl, q5_0, q5_1 (default: f16) (env: LLAMA_ARG_SPEC_DRAFT_CACHE_TYPE_V) |
Sampling params
| Argument | Explanation |
|---|---|
--samplers SAMPLERS |
samplers that will be used for generation in the order, separated by ';' (default: penalties;dry;top_n_sigma;top_k;typ_p;top_p;min_p;xtc;temperature) |
-s, --seed SEED |
RNG seed (default: -1, use random seed for -1) |
--sampler-seq, --sampling-seq SEQUENCE |
simplified sequence for samplers that will be used (default: edskypmxt) |
--ignore-eos |
ignore end of stream token and continue generating (implies --logit-bias EOS-inf) |
--temp, --temperature N |
temperature (default: 0.80) |
--top-k N |
top-k sampling (default: 40, 0 = disabled) (env: LLAMA_ARG_TOP_K) |
--top-p N |
top-p sampling (default: 0.95, 1.0 = disabled) |
--min-p N |
min-p sampling (default: 0.05, 0.0 = disabled) |
--top-nsigma, --top-n-sigma N |
top-n-sigma sampling (default: -1.00, -1.0 = disabled) |
--xtc-probability N |
xtc probability (default: 0.00, 0.0 = disabled) |
--xtc-threshold N |
xtc threshold (default: 0.10, 1.0 = disabled) |
--typical, --typical-p N |
locally typical sampling, parameter p (default: 1.00, 1.0 = disabled) |
--repeat-last-n N |
last n tokens to consider for penalize (default: 64, 0 = disabled) |
--repeat-penalty N |
penalize repeat sequence of tokens (default: 1.00, 1.0 = disabled) |
--presence-penalty N |
repeat alpha presence penalty (default: 0.00, 0.0 = disabled) |
--frequency-penalty N |
repeat alpha frequency penalty (default: 0.00, 0.0 = disabled) |
--dry-multiplier N |
set DRY sampling multiplier (default: 0.00, 0.0 = disabled) |
--dry-base N |
set DRY sampling base value (default: 1.75) |
--dry-allowed-length N |
set allowed length for DRY sampling (default: 2) |
--dry-penalty-last-n N |
set DRY penalty for the last n tokens (default: 64, 0 = disable) |
--dry-sequence-breaker STRING |
add sequence breaker for DRY sampling, clearing out default breakers ('\n', ':', '"', '*') in the process; use "none" to not use any sequence breakers |
--adaptive-target N |
adaptive-p: select tokens near this probability (valid range 0.0 to 1.0; negative = disabled) (default: -1.00) (more info) |
--adaptive-decay N |
adaptive-p: decay rate for target adaptation over time. lower values are more reactive, higher values are more stable. (valid range 0.0 to 0.99) (default: 0.90) |
--dynatemp-range N |
dynamic temperature range (default: 0.00, 0.0 = disabled) |
--dynatemp-exp N |
dynamic temperature exponent (default: 1.00) |
--mirostat N |
use Mirostat sampling. Top K, Nucleus and Locally Typical samplers are ignored if used. (default: 0, 0 = disabled, 1 = Mirostat, 2 = Mirostat 2.0) |
--mirostat-lr N |
Mirostat learning rate, parameter eta (default: 0.10) |
--mirostat-ent N |
Mirostat target entropy, parameter tau (default: 5.00) |
-l, --logit-bias TOKEN_ID(+/-)BIAS |
modifies the likelihood of token appearing in the completion, i.e. --logit-bias 15043+1 to increase likelihood of token ' Hello',or --logit-bias 15043-1 to decrease likelihood of token ' Hello' |
--grammar GRAMMAR |
BNF-like grammar to constrain generations (see samples in grammars/ dir) |
--grammar-file FNAME |
file to read grammar from |
-j, --json-schema SCHEMA |
JSON schema to constrain generations (https://json-schema.org/), e.g. {} for any JSON objectFor schemas w/ external $refs, use --grammar + example/json_schema_to_grammar.py instead |
-jf, --json-schema-file FILE |
File containing a JSON schema to constrain generations (https://json-schema.org/), e.g. {} for any JSON objectFor schemas w/ external $refs, use --grammar + example/json_schema_to_grammar.py instead |
-bs, --backend-sampling |
enable backend sampling (experimental) (default: disabled) (env: LLAMA_ARG_BACKEND_SAMPLING) |
CLI-specific params
| Argument | Explanation |
|---|---|
--server-base URL |
connect to this server instead of starting a new one, example: 'http://localhost:8080' (default: none) |
--verbose-prompt |
print a verbose prompt before generation (default: false) |
--display-prompt, --no-display-prompt |
whether to print prompt at generation (default: true) |
-co, --color [on|off|auto] |
Colorize output to distinguish prompt and user input from generations ('on', 'off', or 'auto', default: 'auto') 'auto' enables colors when output is to a terminal |
-ctxcp, --ctx-checkpoints, --swa-checkpoints N |
max number of context checkpoints to create per slot (default: 32)(more info) (env: LLAMA_ARG_CTX_CHECKPOINTS) |
-cram, --cache-ram N |
set the maximum cache size in MiB (default: 8192, -1 - no limit, 0 - disable)(more info) (env: LLAMA_ARG_CACHE_RAM) |
--context-shift, --no-context-shift |
whether to use context shift on infinite text generation (default: disabled) (env: LLAMA_ARG_CONTEXT_SHIFT) |
-sys, --system-prompt PROMPT |
system prompt to use with model (if applicable, depending on chat template) |
--show-timings, --no-show-timings |
whether to show timing information after each response (default: true) (env: LLAMA_ARG_SHOW_TIMINGS) |
-sysf, --system-prompt-file FNAME |
a file containing the system prompt (default: none) |
-r, --reverse-prompt PROMPT |
halt generation at PROMPT, return control in interactive mode |
-sp, --special |
special tokens output enabled (default: false) |
-cnv, --conversation, -no-cnv, --no-conversation |
whether to run in conversation mode: - does not print special tokens and suffix/prefix - interactive mode is also enabled (default: auto enabled if chat template is available) |
-st, --single-turn |
run conversation for a single turn only, then exit when done will not be interactive if first turn is predefined with --prompt (default: false) |
-mli, --multiline-input |
allows you to write or paste multiple lines without ending each in '' |
--warmup, --no-warmup |
whether to perform warmup with an empty run (default: enabled) |
-mm, --mmproj FILE |
path to a multimodal projector file. see tools/mtmd/README.md note: if -hf is used, this argument can be omitted (env: LLAMA_ARG_MMPROJ) |
-mmu, --mmproj-url URL |
URL to a multimodal projector file. see tools/mtmd/README.md (env: LLAMA_ARG_MMPROJ_URL) |
--mmproj-auto, --no-mmproj, --no-mmproj-auto |
whether to use multimodal projector file (if available), useful when using -hf (default: enabled) (env: LLAMA_ARG_MMPROJ_AUTO) |
--mmproj-offload, --no-mmproj-offload |
whether to enable GPU offloading for multimodal projector (default: enabled) (env: LLAMA_ARG_MMPROJ_OFFLOAD) |
--image, --audio, --video FILE |
path to an image, audio, or video file. use with multimodal models, use comma-separated values for multiple files |
--image-min-tokens N |
minimum number of tokens each image can take, only used by vision models with dynamic resolution (default: read from model) (env: LLAMA_ARG_IMAGE_MIN_TOKENS) |
--image-max-tokens N |
maximum number of tokens each image can take, only used by vision models with dynamic resolution (default: read from model) (env: LLAMA_ARG_IMAGE_MAX_TOKENS) |
-o, --output, --output-file FNAME |
output file (default: '') |
--chat-template-kwargs STRING |
sets additional params for the json template parser, must be a valid json object string, e.g. '{"key1":"value1","key2":"value2"}' (env: LLAMA_ARG_CHAT_TEMPLATE_KWARGS) |
--jinja, --no-jinja |
whether to use jinja template engine for chat (default: enabled) (env: LLAMA_ARG_JINJA) |
--reasoning-format FORMAT |
controls whether thought tags are allowed and/or extracted from the response, and in which format they're returned; one of: - none: leaves thoughts unparsed in message.content- deepseek: puts thoughts in message.reasoning_content- deepseek-legacy: keeps <think> tags in message.content while also populating message.reasoning_content(default: auto) (env: LLAMA_ARG_THINK) |
-rea, --reasoning [on|off|auto] |
Use reasoning/thinking in the chat ('on', 'off', or 'auto', default: 'auto' (detect from template)) (env: LLAMA_ARG_REASONING) |
--reasoning-budget N |
token budget for thinking: -1 for unrestricted, 0 for immediate end, N>0 for token budget (default: -1) (env: LLAMA_ARG_THINK_BUDGET) |
--reasoning-budget-message MESSAGE |
message injected before the end-of-thinking tag when reasoning budget is exhausted (default: none) (env: LLAMA_ARG_THINK_BUDGET_MESSAGE) |
--reasoning-preserve, --no-reasoning-preserve |
preserve reasoning trace in the full history, not just the last assistant message (default: template default) compatible with certain templates having 'supports_preserve_reasoning' capability example: https://docs.z.ai/guides/capabilities/thinking-mode#preserved-thinking (env: LLAMA_ARG_REASONING_PRESERVE) |
--chat-template JINJA_TEMPLATE |
set custom jinja chat template (default: template taken from model's metadata) if suffix/prefix are specified, template will be disabled only commonly used templates are accepted (unless --jinja is set before this flag): list of built-in templates: bailing, bailing-think, bailing2, chatglm3, chatglm4, chatml, command-r, deepseek, deepseek-ocr, deepseek2, deepseek3, exaone-moe, exaone3, exaone4, falcon3, gemma, gigachat, glmedge, gpt-oss, granite, granite-4.0, granite-4.1, grok-2, hunyuan-dense, hunyuan-moe, hunyuan-vl, kimi-k2, llama2, llama2-sys, llama2-sys-bos, llama2-sys-strip, llama3, llama4, megrez, minicpm, mistral-v1, mistral-v3, mistral-v3-tekken, mistral-v7, mistral-v7-tekken, monarch, openchat, orion, pangu-embedded, phi3, phi4, rwkv-world, seed_oss, smolvlm, solar-open, vicuna, vicuna-orca, yandex, zephyr (env: LLAMA_ARG_CHAT_TEMPLATE) |
--chat-template-file JINJA_TEMPLATE_FILE |
set custom jinja chat template file (default: template taken from model's metadata) if suffix/prefix are specified, template will be disabled only commonly used templates are accepted (unless --jinja is set before this flag): list of built-in templates: bailing, bailing-think, bailing2, chatglm3, chatglm4, chatml, command-r, deepseek, deepseek-ocr, deepseek2, deepseek3, exaone-moe, exaone3, exaone4, falcon3, gemma, gigachat, glmedge, gpt-oss, granite, granite-4.0, granite-4.1, grok-2, hunyuan-dense, hunyuan-moe, hunyuan-vl, kimi-k2, llama2, llama2-sys, llama2-sys-bos, llama2-sys-strip, llama3, llama4, megrez, minicpm, mistral-v1, mistral-v3, mistral-v3-tekken, mistral-v7, mistral-v7-tekken, monarch, openchat, orion, pangu-embedded, phi3, phi4, rwkv-world, seed_oss, smolvlm, solar-open, vicuna, vicuna-orca, yandex, zephyr (env: LLAMA_ARG_CHAT_TEMPLATE_FILE) |
--skip-chat-parsing, --no-skip-chat-parsing |
force a pure content parser, even if a Jinja template is specified; model will output everything in the content section, including any reasoning and/or tool calls (default: disabled) (env: LLAMA_ARG_SKIP_CHAT_PARSING) |
--simple-io |
use basic IO for better compatibility in subprocesses and limited consoles |
--log-prompts-dir PATH |
Log prompts to directory (auto-created if not present; only used for debugging, default: disabled) |
--spec-draft-hf, -hfd, -hfrd, --hf-repo-draft <user>/<model>[:quant] |
Same as --hf-repo, but for the draft model (default: unused) (env: LLAMA_ARG_SPEC_DRAFT_HF_REPO) |
--spec-draft-threads, -td, --threads-draft N |
number of threads to use during generation (default: same as --threads) |
--spec-draft-threads-batch, -tbd, --threads-batch-draft N |
number of threads to use during batch and prompt processing (default: same as --threads-draft) |
--spec-draft-cpu-mask, -Cd, --cpu-mask-draft M |
Draft model CPU affinity mask. Complements cpu-range-draft (default: same as --cpu-mask) |
--spec-draft-cpu-range, -Crd, --cpu-range-draft lo-hi |
Ranges of CPUs for affinity. Complements --cpu-mask-draft |
--spec-draft-cpu-strict, --cpu-strict-draft <0|1> |
Use strict CPU placement for draft model (default: same as --cpu-strict) |
--spec-draft-prio, --prio-draft N |
set draft process/thread priority : 0-normal, 1-medium, 2-high, 3-realtime (default: 0) |
--spec-draft-poll, --poll-draft <0|1> |
Use polling to wait for draft model work (default: same as --poll) |
--spec-draft-cpu-mask-batch, -Cbd, --cpu-mask-batch-draft M |
Draft model CPU affinity mask. Complements cpu-range-draft (default: same as --cpu-mask) |
--spec-draft-cpu-strict-batch, --cpu-strict-batch-draft <0|1> |
Use strict CPU placement for draft model (default: --cpu-strict-draft) |
--spec-draft-prio-batch, --prio-batch-draft N |
set draft process/thread priority : 0-normal, 1-medium, 2-high, 3-realtime (default: 0) |
--spec-draft-poll-batch, --poll-batch-draft <0|1> |
Use polling to wait for draft model work (default: --poll-draft) |
--spec-draft-override-tensor, -otd, --override-tensor-draft <tensor name pattern>=<buffer type>,... |
override tensor buffer type for draft model |
--spec-draft-cpu-moe, -cmoed, --cpu-moe-draft |
keep all Mixture of Experts (MoE) weights in the CPU for the draft model (env: LLAMA_ARG_SPEC_DRAFT_CPU_MOE) |
--spec-draft-n-cpu-moe, --spec-draft-ncmoe, -ncmoed, --n-cpu-moe-draft N |
keep the Mixture of Experts (MoE) weights of the first N layers in the CPU for the draft model (env: LLAMA_ARG_SPEC_DRAFT_N_CPU_MOE) |
--spec-draft-n-max N |
number of tokens to draft for speculative decoding (default: 3) (env: LLAMA_ARG_SPEC_DRAFT_N_MAX) |
--spec-draft-n-min N |
minimum number of draft tokens to use for speculative decoding (default: 0) (env: LLAMA_ARG_SPEC_DRAFT_N_MIN) |
--spec-draft-p-split, --draft-p-split P |
speculative decoding split probability (default: 0.10) (env: LLAMA_ARG_SPEC_DRAFT_P_SPLIT) |
--spec-draft-p-min, --draft-p-min P |
minimum speculative decoding probability (greedy) (default: 0.00) (env: LLAMA_ARG_SPEC_DRAFT_P_MIN) |
--spec-draft-backend-sampling, --no-spec-draft-backend-sampling |
offload draft sampling to the backend (default: enabled) (env: LLAMA_ARG_SPEC_DRAFT_BACKEND_SAMPLING) |
--spec-draft-device, -devd, --device-draft <dev1,dev2,..> |
comma-separated list of devices to use for offloading the draft model (none = don't offload) use --list-devices to see a list of available devices |
--spec-draft-ngl, -ngld, --gpu-layers-draft, --n-gpu-layers-draft N |
max. number of draft model layers to store in VRAM, either an exact number, 'auto', or 'all' (default: auto) (env: LLAMA_ARG_N_GPU_LAYERS_DRAFT) |
--spec-draft-model, -md, --model-draft FNAME |
draft model for speculative decoding (default: unused) (env: LLAMA_ARG_SPEC_DRAFT_MODEL) |
--spec-type none,draft-simple,draft-eagle3,draft-mtp,draft-dflash,draft-dspark,ngram-simple,ngram-map-k,ngram-map-k4v,ngram-mod,ngram-cache |
comma-separated list of types of speculative decoding to use (default: none) (env: LLAMA_ARG_SPEC_TYPE) |
--spec-ngram-mod-n-min N |
minimum number of ngram tokens to use for ngram-based speculative decoding (default: 48) |
--spec-ngram-mod-n-max N |
maximum number of ngram tokens to use for ngram-based speculative decoding (default: 64) |
--spec-ngram-mod-n-match N |
ngram-mod lookup length (default: 24) |
--spec-ngram-simple-size-n N |
ngram size N for ngram-simple speculative decoding, length of lookup n-gram (default: 12) |
--spec-ngram-simple-size-m N |
ngram size M for ngram-simple speculative decoding, length of draft m-gram (default: 48) |
--spec-ngram-simple-min-hits N |
minimum hits for ngram-simple speculative decoding (default: 1) |
--spec-ngram-map-k-size-n N |
ngram size N for ngram-map-k speculative decoding, length of lookup n-gram (default: 12) |
--spec-ngram-map-k-size-m N |
ngram size M for ngram-map-k speculative decoding, length of draft m-gram (default: 48) |
--spec-ngram-map-k-min-hits N |
minimum hits for ngram-map-k speculative decoding (default: 1) |
--spec-ngram-map-k4v-size-n N |
ngram size N for ngram-map-k4v speculative decoding, length of lookup n-gram (default: 12) |
--spec-ngram-map-k4v-size-m N |
ngram size M for ngram-map-k4v speculative decoding, length of draft m-gram (default: 48) |
--spec-ngram-map-k4v-min-hits N |
minimum hits for ngram-map-k4v speculative decoding (default: 1) |
--draft, --draft-n, --draft-max N |
the argument has been removed. use --spec-draft-n-max or --spec-ngram-mod-n-max (env: LLAMA_ARG_DRAFT_MAX) |
--draft-min, --draft-n-min N |
the argument has been removed. use --spec-draft-n-min or --spec-ngram-mod-n-min (env: LLAMA_ARG_DRAFT_MIN) |
--gpt-oss-20b-default |
use gpt-oss-20b (note: can download weights from the internet) |
--gpt-oss-120b-default |
use gpt-oss-120b (note: can download weights from the internet) |
--vision-gemma-4b-default |
use Gemma 3 4B QAT (note: can download weights from the internet) |
--vision-gemma-12b-default |
use Gemma 3 12B QAT (note: can download weights from the internet) |
--spec-default |
enable default speculative decoding config |