vrr/koboldcpp
2
0
Fork 0
mirror of https://github.com/LostRuins/koboldcpp.git synced 2026-08-27 01:04:19 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

server: refactor + correctness fixes for metrics (#26920)

Browse source 
* server: refactor metrics

* move most fields to server_slot_stats

* cont

* rm result_timings

* tie stats to batch

* cont

* nits: move place in code

* exclude first generated token

* more accurate batch metrics tracking

* n_predict --> n_gen

* metrics_on_prediction

* metrics_flush_idle

* metrics: seperate cache/processed prompt tokens

* refactor server_task_result_metrics

* add test

* nits

* fix flush before reset()

* cont

* rm dead code

* nits
This commit is contained in:
Xuan-Son Nguyen 2026-08-13 10:02:01 +02:00 • committed by GitHub
parent e79e4bf660
commit decaf508bb
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
6 changed files with 813 additions and 475 deletions
Download patch file Download diff file Expand all files Collapse all files

27
tools/server/server-common.cpp
View file

@ -60,6 +60,33 @@ json format_error_response(const std::string & message, const enum error_type ty
};
}
//
// server_slot_stats
//
json server_slot_stats::to_json() const {
json base = {
{"cache_n", n_prompt_cached},
{"prompt_n", n_prompt_processed},
{"prompt_ms", t_prompt_ms()},
{"prompt_per_token_ms", t_prompt_per_token_ms()},
{"prompt_per_second", n_prompt_tps()},
{"predicted_n", n_gen},
{"predicted_ms", t_gen_ms()},
{"predicted_per_token_ms", t_gen_per_token_ms()},
{"predicted_per_second", n_gen_tps()},
};
if (n_draft_tokens > 0) {
base["draft_n"] = n_draft_tokens;
base["draft_n_accepted"] = n_draft_accepted;
}
return base;
}
//
// random string / id
//