vrr/prima.cpp
2
0
Fork 0
mirror of https://github.com/Lizonghang/prima.cpp.git synced 2025-09-06 04:19:15 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions
4204 commits 6 branches 0 tags 73 MiB
Commit graph

4204 commits

This branch
This branch
All branches
Author SHA1 Message Date
Zonghang Li
dd589561b4 improve the computing buffer estimate 2025-06-19 08:02:43 +00:00
Li, Zonghang
0b4ffdfde5 Merge branch 'dev' 2025-06-17 09:40:27 +04:00
DeEMO
deeec668b8 fix: n_worker in draft model 
(cherry picked from commit 921ad2b453b24b715ad5db6a703fb3df65fdcb80)
 2025-06-17 13:23:20 +08:00
Zonghang Li
2b902f89bd
fix: change default ip to 127.0.0.1 & improve args for setting ports 
fix: change default ip to `127.0.0.1` & improve args for setting ports
 2025-06-17 08:23:25 +04:00
DeEMO
67c4f70357 fix: add log when serving as a proxy 2025-06-17 12:08:53 +08:00
DeEMO
6ff38b2a0c add args: data-port and signal-port 2025-06-17 12:00:04 +08:00
DeEMO
104e3b2356 fix: replace localhost to 127.0.0.1 2025-06-17 11:27:58 +08:00
Li, Zonghang
fbbc30c950 Merge branch 'speculative' into dev 2025-06-16 13:27:36 +04:00
Zonghang Li
dc797e94f5
Fix speculative decoding 
Power prima.cpp with speculative decoding: Further speeds up by up to 80%
 2025-06-16 12:11:12 +04:00
Li, Zonghang
dfb1feb54e update README 2025-06-16 12:09:07 +04:00
Li, Zonghang
45de284f3d Merge branch 'fix' into speculative 2025-06-14 18:57:17 +04:00
Li, Zonghang
f38cfc625c Merge branch 'fix' into dev 2025-06-14 18:56:36 +04:00
Li, Zonghang
b5ccd62135 fix n_gpu_layers allocation errors 2025-06-14 18:55:53 +04:00
Li, Zonghang
0a535cbdc1 Merge branch 'speculative' of github.com:Lizonghang/prima.cpp into speculative 2025-06-13 13:31:12 +04:00
Li, Zonghang
c9cae626cf speculative: free sockets and send stop signal when inference ends 2025-06-13 13:30:29 +04:00
Li, Zonghang
2687ef3126 speculative: free sockets and send stop signal when inference ends 2025-06-13 11:25:42 +04:00
Li, Zonghang
dc875bbef9 fix speculative decoding 2025-06-13 08:18:12 +04:00
Zonghang Li
ba29717613
add feature: keep the forwarder if its previous device cannot directly connect to its next device. 
feat: nodes attempt connections during topology rebuild while preserving forwarders
 2025-06-12 16:57:35 +04:00
DeEMO
d4618de991 fix: block when free socket 2025-06-12 12:26:10 +00:00
DeEMO
2039e3b0c1 fix: send and recv meta 2025-06-12 12:26:10 +00:00
DeEMO
d6c8d322cd fix try_connect 2025-06-12 12:26:10 +00:00
DeEMO
d1b97f798e support reconnection 2025-06-12 12:26:09 +00:00
Zonghang Li
e50b3aa473
Merge pull request #27 from Lizonghang/lizh_dev 
Fix seq_id mismatch between the head and worker devices.
 2025-06-11 17:12:08 +04:00
Li, Zonghang
3e6d831930 fix seq_id mismatch between head and worker devices 2025-06-11 17:10:21 +04:00
Li, Zonghang
fb9b1f2b00 reformat llama.cpp 2025-06-09 13:04:22 +04:00
Li, Zonghang
fbf853341b add endpoint /v1/cancel 2025-06-07 11:34:38 +04:00
Zonghang Li
c8af1be27e
Merge pull request #24 from Lizonghang/lizh_dev 
Fix batch decoding and dynamic batching.
 2025-06-07 01:02:10 +04:00
Li, Zonghang
22a6ddef13 fix batch decoding and dynamic batching 2025-06-07 00:53:56 +04:00
Lizonghang
e56be76bdf assume only a single seq_id per token is needed 2025-06-07 00:42:44 +04:00
Lizonghang
d8aea899d1 fix n_seq_id and seq_id 2025-06-06 23:58:03 +04:00
Lizonghang
a1a2238831 add batch_all.n_seq_id and batch_all.seq_id to sync_meta 2025-06-06 23:36:53 +04:00
Lizonghang
68ecc8509d add batch_all.logits to sync_meta 2025-06-06 22:58:48 +04:00
Lizonghang
500e066a2f fix batch decoding and dynamic batching 2025-06-06 16:53:22 +04:00
Lizonghang
ef1e10101e add test for IQ1 and doc for device selection 2025-06-04 15:12:00 +04:00
Lizonghang
27756ee182 fix: enable rolling back set assignment when all devices are assigned to M4 but no feasible solutions 2025-06-04 15:11:29 +04:00
Li, Zonghang
6439090920 reformat code 2025-06-03 23:53:24 +04:00
Li, Zonghang
b6fdbd541b Merge branch 'dev' of github.com:Lizonghang/prima.cpp into dev 2025-06-03 18:20:17 +04:00
Lizonghang
9f0ec78a4b Merge branch 'dev' of github.com:Lizonghang/prima.cpp into dev 2025-06-03 18:18:53 +04:00
Li, Zonghang
a01fafd126 Merge branch 'main' into dev 2025-06-03 17:56:47 +04:00
Li, Zonghang
1b3b6a506f fix: add warm-up in profiling to prevent init delay 2025-06-03 17:10:09 +04:00
Li, Zonghang
b30f749e5e fix n_embd cannot be divided by quantized block size 2025-06-03 14:06:31 +04:00
Li, Zonghang
e25c739ecf
Merge pull request #19 from yezhizi/feat/auto-exit 
feat: ranks with only 1 layer auto-exit and rebuild topology
 2025-05-20 02:04:50 +08:00
Li, Zonghang
7b0ededd24
Merge branch 'dev' into feat/auto-exit 2025-05-20 02:04:14 +08:00
Lizonghang
421b3deca5 fix llama-cli pos sync 2025-05-19 18:08:27 +04:00
Lizonghang
c54a6a0132 fix context shifting 2025-05-19 16:58:35 +04:00
DeEMO
34eaa8224d fix: handle socket closure and connection in llama_rebuild_topo 
Signed-off-by: DeEMO <yzzxrx@gmail.com>
 2025-05-19 09:22:35 +00:00
DeEMO
8b61cb2fa4 fix: adapt the new topo 
Signed-off-by: DeEMO <yzzxrx@gmail.com>
 2025-05-19 09:22:29 +00:00
DeEMO
df16b1876f refactor: add zmq helper to generate message 
Signed-off-by: DeEMO <yzzxrx@gmail.com>
 2025-05-19 09:22:24 +00:00
DeEMO
0ad009a2f4 fix: update serialization and deserialization for next_ip in device_info 
Signed-off-by: DeEMO <yzzxrx@gmail.com>
 2025-05-19 09:22:16 +00:00
DeEMO
4b36aef157 fix some bugs 
Signed-off-by: DeEMO <yzzxrx@gmail.com>
 2025-05-19 09:22:08 +00:00