mirror of
https://github.com/LostRuins/koboldcpp.git
synced 2026-10-03 11:35:46 +00:00
7d6f5d02bb
* model : add support for HrmTextForCausalLM (DFM Mimir 1B)
HRM-Text runs two transformer stacks (low, high) in an alternating cycle over the same token stream. The low-cycle state z_l starts from a learned [n_embd] tensor and is broadcast over positions.
- conversion: new writer for the fused gqkv projection (order gate,q,k,v) remapped to llama.cpp q/k/v plus a separate sigmoid gate tensor
- loader: block_count = lps * h_cycles * (l_cycles + 1) cache slots aliasing 2*lps physical blocks via struct copies
- graph: looped build with sigmoid-gated attention, SwiGLU FFN and parameterless RMS norms; learned embedding_scale applied in build_inp_embd
- saver: pointer-deduplicated layer loop (looped archs alias tensors)
- tests: hrm_text fixture (lps 1, h 2, l 3) in test-llama-archs
Limitations:
causal attention only - the upstream prefix-LM mode is not implemented (the prefix_lm GGUF key round-trips unused).
The KV cache holds one entry per pass: 128 layers for Mimir 1B, i.e. 4x a same-width 32-layer model - about 3072 MiB at ctx 4096 in F16 (halves with q8_0 KV + FA).
Every token runs all 128 block passes, so decode cost is roughly 4x a dense model of equal width (2.65 t/s BF16, 8-thread desktop CPU).
Verified against the HF reference: identical argmax at 334/334 positions across 20 prompts (BF16 GGUF vs FP32 golden).
q8_0 requant: 95.8% top-1, all remaining misses inside the HF top-5 (accumulated error over 128 sequential blocks).
AI usage disclosure: YES
Used GLM-5.3 for the majority of code AI-generated under my direction, all gates verified locally.
All in all I could say that I have written less than 20% of the code and most of the heavy lifting has been done by the model. As such, this should be considered experimental.
* Update conversion/hrm_text.py
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* Update src/llama-arch.cpp
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* convert : add gguf_writer methods for hrm_text metadata
replace raw add_uint32/add_bool calls with dedicated GGUFWriter methods, following the add_embedding_scale pattern
Assisted-by: GLM-5.3
* convert : map regular hrm_text tensors via tensor_mapping
delegate unfused checkpoints to the base tensor mapping; training-style attn. names are renamed to self_attn. so the patterns match
Assisted-by: GLM-5.3
* model : format hrm-text build_* calls as in other models
one argument group per line, matching sibling model files
Assisted-by: GLM-5.3
* llama : move hrm z_l_init table entries out of the nemotron group
place the name and tensor-info entries with the other global input tensors
Assisted-by: GLM-5.3
* convert : slim down hrm_text comments
Assisted-by: GLM-5.3
* convert : build hrm_text block tensor names from the {bid} template
The tensor map holds concrete per-block names, so format the template
with the computed layer index before handing it to super().
* llama : name hrm metadata keys in their own hrm. namespace
The four keys are arch-independent, unlike the arch-substituted
Keys.LLM entries, so group them under Keys.HRM (like Keys.Split) and
rename the llm_kv entries to LLM_KV_HRM_*. Only our own GGUFs carry
the old hrm_text.* keys; they are regenerated.
* Update src/llama-model-saver.cpp
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* llama : keep hrm metadata keys arch-substituted
Per review: the GGUF keys stay "{arch}.h_cycles" style, so the Python
members drop the LLM_KV_HRM_ prefix and keep arch templates; C++ keeps
the LLM_KV_HRM_* enums. GGUF output is unchanged - existing files and
HF uploads stay valid.
* Update gguf-py/gguf/constants.py
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* Update src/llama-arch.cpp
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* Update src/llama-arch.cpp
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* convert : rename hrm writer methods to add_hrm_*
Generic names like add_h_cycles/add_prefix_lm are too broad on the
shared GGUFWriter; prefix them with hrm_ like the metadata keys.
* model : fix meta-split lookup for archs with aliased cache slots
Cache tensors of archs that alias physical blocks across looped slots
(hrm_text, nanbeige with num_loops > 1) can reference block indices
without weight tensor names. Take the output projection from the layer
array instead of asserting; all other lookups are unchanged.
* model : replicate hrm_text tensors on meta devices instead of splitting
The aliased cache slots rotate split states differently from their
physical weights, so the meta-split execution invariants (set_rows
requires the cache state to match the token indices) cannot hold for
any device count. Replicate all hrm_text tensors on every meta device
instead; single-device and non-meta paths are unchanged.
Assisted-by: Claude Sonnet
---------
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
|
||
|---|---|---|
| .. | ||
| __init__.py | ||
| afmoe.py | ||
| arctic.py | ||
| baichuan.py | ||
| bailingmoe.py | ||
| bailingmoe3.py | ||
| base.py | ||
| bert.py | ||
| bitnet.py | ||
| bloom.py | ||
| chameleon.py | ||
| chatglm.py | ||
| codeshell.py | ||
| cogvlm.py | ||
| command_r.py | ||
| dbrx.py | ||
| deci.py | ||
| deepseek.py | ||
| dots1.py | ||
| dots3.py | ||
| dotsocr.py | ||
| dream.py | ||
| ernie.py | ||
| exaone.py | ||
| falcon.py | ||
| falcon_h1.py | ||
| gemma.py | ||
| glm.py | ||
| gpt2.py | ||
| gpt_oss.py | ||
| gptneox.py | ||
| granite.py | ||
| grok.py | ||
| grovemoe.py | ||
| hrm_text.py | ||
| hunyuan.py | ||
| hy_v4.py | ||
| internlm.py | ||
| internvl.py | ||
| jais.py | ||
| jamba.py | ||
| januspro.py | ||
| kimi_k3.py | ||
| kimi_linear.py | ||
| kimivl.py | ||
| laguna.py | ||
| lfm2.py | ||
| lighton_ocr.py | ||
| llada.py | ||
| llama.py | ||
| llama4.py | ||
| llava.py | ||
| maincoder.py | ||
| mamba.py | ||
| maple.py | ||
| mellum.py | ||
| mimo.py | ||
| minicpm.py | ||
| minimax.py | ||
| mistral.py | ||
| mistral3.py | ||
| mpt.py | ||
| muse_glimmer.py | ||
| nanbeige.py | ||
| nemotron.py | ||
| olmo.py | ||
| openelm.py | ||
| orion.py | ||
| pangu.py | ||
| phi.py | ||
| pixtral.py | ||
| plamo.py | ||
| plm.py | ||
| pockettts.py | ||
| qwen.py | ||
| qwen3tts.py | ||
| qwen3vl.py | ||
| qwen4exp.py | ||
| qwenvl.py | ||
| refact.py | ||
| rwkv.py | ||
| sarashina2.py | ||
| smallthinker.py | ||
| smolvlm.py | ||
| spark2_5.py | ||
| stablelm.py | ||
| starcoder.py | ||
| step3.py | ||
| t5.py | ||
| talkie.py | ||
| ultravox.py | ||
| wavtokenizer.py | ||
| xverse.py | ||
| youtuvl.py | ||