mirror of
https://github.com/LostRuins/koboldcpp.git
synced 2026-10-03 19:45:41 +00:00
0ba6499c3b
* CUDA: Allow CUDA optimization per split for multi-GPU. Previous guard caused multi-GPU to skip the graph optimization. The graph is already split per device and the optimization doesnt run over the whole model but once per split, and thus should be allowed. However, the CUDA event ggml_cuda_concurrent_event belongs to whichever GPU was "current" when created. If the pass ran while GPU 0 was current, it would stick and during event creation for the second GPU it would land on GPU 0. The fix: set the device explicitly ggml_cuda_set_device(cuda_ctx->device); Default behaviour remains unchanged, only active for GGML_CUDA_GRAPH_OPT=1. Explicit device setting pattern re-used from ggml_backend_cuda_graph_compute. * Update ggml/src/ggml-cuda/ggml-cuda.cu Co-authored-by: Aman Gupta <amangupta052@gmail.com> --------- Co-authored-by: tannerbruhn <tannerbruhn@users.noreply.github.com> Co-authored-by: Aman Gupta <amangupta052@gmail.com> |
||
|---|---|---|
| .. | ||
| cmake | ||
| include | ||
| src | ||
| .gitignore | ||
| CMakeLists.txt | ||