Piotr Wilkin (ilintar) f4e276a206 ggml-cuda : convert contiguous tensors four elements at a time (#29155) ... convert_unary handles the contiguous case through the general strided kernel, one element per thread: each lane reads 4 bytes and writes 2. Converting the activations for a bf16 matrix multiplication that way moves 126 MB in 1021 us on gfx1151, about 65% of what the memory system can do. Give the contiguous path its own kernel that takes four elements per thread through a vector type, so a warp loads 512 bytes at a time instead of 128. It is used only when the element count is a multiple of four and both pointers carry the alignment the vector type needs, and falls back to the strided kernel otherwise. Model level, Qwen3.8-Next-Flash IQ3_XXS on gfx1151, llama-bench -ub 2048 -r 6, mean of the last 3 reps, ABBA counterbalanced: pp2048 688.0 680.0 -> 694.3 691.1 +1.26% tg128 24.8 24.8 -> 24.8 24.8 +0.14% Every conversion in a prefill takes the new kernel (kernel trace: 1146 convert_unary_cont_vec4, no convert_unary). Output is bit identical; MUL_MAT, MUL_MAT_ID, CPY, CONT, GET_ROWS and SET_ROWS pass. Assisted-by: Claude Opus 5		2026-09-21 18:00:51 +02:00
..
cmake	CUDA: replace GGML_FA_ALL_QUANTS with GGML_FA_QUANTS, more control over what is compiled (#28079)	2026-09-09 12:50:08 +02:00
include	sycl : pinned memory use right device context instead of 0 (#28895)	2026-09-21 13:58:59 +03:00
src	ggml-cuda : convert contiguous tensors four elements at a time (#29155)	2026-09-21 18:00:51 +02:00
.gitignore
CMakeLists.txt	ggml : bump version to 0.24.0 (ggml/1627)	2026-09-14 16:45:33 +03:00