vrr/koboldcpp
2
0
Fork 0
mirror of https://github.com/LostRuins/koboldcpp.git synced 2026-07-10 01:18:32 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions
koboldcpp/tests
History
Piotr Wilkin (ilintar) 75a48a9055
cuda: enable topk-moe fusion for 288 experts (#25267) 
* cuda: enable topk-moe fusion for 288 experts

The topk-moe fusion only accepted power-of-2 expert counts (or the
special-cased 576), so models with 288 experts (e.g. Step-3.7-Flash)
fell back to the unfused per-layer routing chain: softmax/sigmoid,
argsort, get_rows, sum_rows, div, clamp, scale. At batch size 1 that
is ~330 extra tiny graph nodes per token.

288 is a multiple of the warp size, so the existing kernel already
handles it; this adds the missing template instantiation and accepts
288 in the eligibility check.

Measured on gfx1151 with Step-3.7-Flash IQ4_XS (llama-bench,
-b 4096 -ub 4096 -fa 1 -dio 1 -ctk q8_0 -ctv q8_0; machine idle,
before/after paired so pp4096 stays matched as a load control):

  test            | before         | after
  ----------------+----------------+----------------
  pp4096          | 460.99 ± 0.45  | 462.47 ± 0.34   (unchanged)
  tg128           |  19.10 ± 0.04  |  19.56 ± 0.03   (+2.4%)
  tg128 @ d30000  |  12.68 ± 0.04  |  12.69 ± 0.03   (unchanged)

Prompt processing is unaffected (the fusion only touches decode
routing). The decode gain is ~+2.4% at shallow context and fades with
depth: by 30k tokens each step is attention-bound over the KV cache,
so removing the fixed routing overhead is no longer visible.

Assisted-By: Claude Fable 5 <noreply@anthropic.com>

* Update tests/test-backend-ops.cpp

Co-authored-by: Oliver Simons <osimons@nvidia.com>

* Add comment for case 288 in topk-moe.cu

---------

Co-authored-by: Oliver Simons <osimons@nvidia.com>
2026-07-03 15:36:55 +02:00
..
peg-parser common/peg : implement ac parser for stricter grammar generation (#24869) 2026-06-21 16:20:58 -05:00
snapshots tests : add unit test coverage for llama_tensor_get_type (#20112) 2026-04-02 22:53:58 +02:00
.gitignore tests : add unit test coverage for llama_tensor_get_type (#20112) 2026-04-02 22:53:58 +02:00
CMakeLists.txt common : remove unused regex-partial (#25118) 2026-06-29 08:48:39 +02:00
get-model.cpp ci : add model tests + script wrapper (#4586) 2024-01-26 14:18:00 +02:00
get-model.h ci : add model tests + script wrapper (#4586) 2024-01-26 14:18:00 +02:00
gguf-model-data.cpp ci : add [no release] keyword + fix sanitizer builds (#23728) 2026-05-26 19:05:48 +03:00
gguf-model-data.h tests : add unit test coverage for llama_tensor_get_type (#20112) 2026-04-02 22:53:58 +02:00
test-alloc.cpp chore : correct typos [no ci] (#20041) 2026-03-05 08:50:21 +01:00
test-arg-parser.cpp arg: try fixing test-args-parser randomly fails (#24826) 2026-06-20 19:45:27 +02:00
test-autorelease.cpp docs : Minor cleanups (#19252) 2026-02-02 08:38:55 +02:00
test-backend-ops.cpp cuda: enable topk-moe fusion for 288 experts (#25267) 2026-07-03 15:36:55 +02:00
test-backend-sampler.cpp llama : deprecate llama_set_warmup (#24009) 2026-06-02 10:30:38 +03:00
test-barrier.cpp Fix race conditions in threadpool when dealing with dynamic/frequent n_threads changes (#17748) 2025-12-10 12:32:23 -08:00
test-c.c ggml : remove kompute backend (#14501) 2025-07-03 07:48:32 +03:00
test-chat-auto-parser.cpp chat: fix whitespace problems once and for all (#24624) 2026-06-15 08:27:10 +02:00
test-chat-peg-parser.cpp common/parser: add proper reasoning tag prefill reading (#20424) 2026-03-19 16:58:21 +01:00
test-chat-template.cpp jinja: add --dump-prog for debugging (#25086) 2026-06-28 15:50:31 +02:00
test-chat.cpp chat : implement minicpm5 parser (#24889) 2026-06-28 16:53:32 +02:00
test-col2im-1d.cpp ggml : add GGML_OP_COL2IM_1D (#24206) 2026-06-09 12:01:37 +03:00
test-double-float.cpp ggml : minor naming changes (#8433) 2024-07-12 10:46:02 +03:00
test-export-graph-ops.cpp binaries : Improve rpc-server and export-graph-ops names. (#25045) 2026-06-27 10:31:29 +03:00
test-gbnf-validator.cpp cmake : do not include ./src as public for libllama (#13062) 2025-04-24 16:00:10 +03:00
test-gguf-model-data.cpp tests : add unit test coverage for llama_tensor_get_type (#20112) 2026-04-02 22:53:58 +02:00
test-gguf.cpp ggml: gguf_init_from_callback and gguf_init_from_buffer (#22341) 2026-05-25 11:33:29 +02:00
test-grammar-integration.cpp common/grammar: fix grammar parsing issues to prevent stack overflow and hangs (#18604) 2026-03-21 18:43:35 +01:00
test-grammar-llguidance.cpp tool/ex/tests: consistently free ctx, then model (#18168) 2025-12-22 11:00:37 +01:00
test-grammar-parser.cpp common/grammar: fix grammar parsing issues to prevent stack overflow and hangs (#18604) 2026-03-21 18:43:35 +01:00
test-jinja.cpp chat : implement minicpm5 parser (#24889) 2026-06-28 16:53:32 +02:00
test-json-schema-to-grammar.cpp common/json-schema-to-grammar : align spacing rules with parsers (#24835) 2026-06-20 17:43:04 -05:00
test-llama-archs.cpp DeepSeek V4 (#24162) 2026-06-29 16:58:51 +08:00
test-llama-grammar.cpp common/grammar: fix grammar parsing issues to prevent stack overflow and hangs (#18604) 2026-03-21 18:43:35 +01:00
test-log.cpp common: Intentionally leak logger instance to fix hanging on Windows (#22273) 2026-04-29 10:58:43 +03:00
test-lora-conversion-inference.sh cli: new CLI experience (#17824) 2025-12-10 15:28:59 +01:00
test-model-load-cancel.cpp llama : update llama_model API names (#11063) 2025-01-06 10:55:18 +02:00
test-mtmd-c-api.c mtmd : add video input support (#24269) 2026-06-08 14:40:12 +03:00
test-opt.cpp tests : fix test-opt with GGML_BACKEND_DL (#15599) 2025-08-26 22:14:38 +02:00
test-peg-parser.cpp Autoparser - complete refactoring of parser architecture (#18675) 2026-03-06 21:01:00 +01:00
test-quant-type-selection.cpp tests : add unit test coverage for llama_tensor_get_type (#20112) 2026-04-02 22:53:58 +02:00
test-quantize-fns.cpp ggml-cpu: fix SVE leftover path in ggml_vec_dot_f32 (#24699) 2026-06-26 10:41:56 +03:00
test-quantize-perf.cpp ci: run the x64 and arm ci on the github machines instead (#16183) 2025-09-25 08:06:06 +03:00
test-quantize-stats.cpp libs : rename libcommon -> libllama-common (#21936) 2026-04-17 11:11:46 +03:00
test-reasoning-budget.cpp common : support manually triggering the reasoning budget end sequence (#23949) 2026-06-01 11:37:11 +02:00
test-recurrent-state-rollback.cpp llama + spec: MTP Support (#22673) 2026-05-16 20:06:23 +08:00
test-rope.cpp ggml-cpu: templateify ggml_compute_forward_rope_f32 and _f16 (#16805) 2025-11-11 13:33:24 +02:00
test-sampling.cpp sampling : remove unconditional softmax+sort in top-n-sigma sampler (#22645) 2026-06-22 14:08:32 +03:00
test-save-load-state.cpp tests : refactor test-save-load-state to accept token input (#24073) 2026-06-04 08:06:36 +03:00
test-state-restore-fragmented.cpp common : only load backends when required (#22290) 2026-05-05 09:23:50 +02:00
test-thread-safety.cpp tests : synchronize contexts at end of test-thread-safety (#24935) 2026-06-25 09:22:51 +03:00
test-tokenizer-0.cpp tool/ex/tests: consistently free ctx, then model (#18168) 2025-12-22 11:00:37 +01:00
test-tokenizer-0.py requirements : update transformers to 5.5.1 (#21617) 2026-04-09 12:36:29 +02:00
test-tokenizer-0.sh model : add Jina Embeddings v5 Nano (partial EuroBERT) support (#19826) 2026-02-26 12:14:09 +01:00
test-tokenizer-1-bpe.cpp tool/ex/tests: consistently free ctx, then model (#18168) 2025-12-22 11:00:37 +01:00
test-tokenizer-1-spm.cpp tool/ex/tests: consistently free ctx, then model (#18168) 2025-12-22 11:00:37 +01:00
test-tokenizer-random.py requirements : update transformers to 5.5.1 (#21617) 2026-04-09 12:36:29 +02:00
test-tokenizers-repo.sh devops: add s390x & ppc64le CI (#15925) 2025-09-27 02:03:33 +08:00
testing.h common : implement new jinja template engine (#18462) 2026-01-16 11:22:06 +01:00