Commit graph

2 commits

Author SHA1 Message Date
Alessandro
2afffe8edc Overlap embedding warm-up with chat restoration
Some checks are pending
Build And Publish Docker Images / plan (push) Waiting to run
Build And Publish Docker Images / build (push) Blocked by required conditions
Load sentence-transformers on the existing cache miss and construct the
preloaded model off the shared background event loop. Start warm-up before
chat restoration inside extensible init_a0, preserving start-hook control
and sharing the loaded model with ordinary runtime requests.

Remove unused speech preload imports and dead warm-up functions. Seed APT
indexes only when absent, skipping the refresh on normal container boots.
Keep the existing Python tunnel service and its provider lifecycle.

Validation: 49 focused framework tests passed, including extension-hook
short-circuit behavior, concurrent embedding-cache reuse, APT cache states,
tunnel behavior and CSRF origins. Three cold Fable chat trials per version:
median submit-to-first-token 9.588s -> 4.234s;
local preparation 6.904s -> 1.197s;
HTTP readiness 42.637s -> 41.139s;
boot-to-first-token 52.093s -> 46.657s.
Provider timing varies; these are local repeated measurements. The live
baseline already omitted APT refresh, so it receives no timing credit.
2026-10-10 16:36:08 +02:00
Alessandro
5a96eda078 Reuse preloaded local embedding models
Cache the underlying SentenceTransformer by its effective constructor options so startup preload and first-chat initialization share one instance while wrappers keep their own runtime configuration. Pass the runtime embedding configuration through preload and cover reuse, concurrency, option changes, and failure handling.
2026-08-26 23:04:32 +02:00