51 commits

Author	SHA1	Message	Date
Daniel Han	2affd53e1f	Studio: recall the latest version of a fact, not the most quotable one (#9161) ... * Studio: add rolling context windows for local GGUF chat * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: return rolling context metadata for non-stream chats * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: keep original context when rolling fit fails * Studio: keep instruction groups independently protected * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: preserve rolling context metadata across retries * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: count sanitized rolling context prompts * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: refit rolling context after respawn * Studio: scope middle truncation to passthrough * Studio: refit tool prompts after respawn * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: retain later choice truncation metadata * Studio: report clipping-only context truncation * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: expose the turns a rolling fit evicts Rolling context eviction currently drops turns with no way for a caller to learn which ones went. Extract the turn grouping so the eviction unit is reusable, add an identity-based diff of what a fit removed, and let a caller reserve room for content it intends to add back after fitting. The reserve deliberately does not affect whether trimming happens at all, only how far it goes once it is already required, so a conversation that fits today is still returned untouched. * Studio: archive evicted turns into a per-thread rag scope An evicted turn is currently gone for the rest of the session, so the model will state the conversation began wherever its visible context begins. Keep the turns the rolling window drops in a searchable scope built on the existing store, chunker, embedder and hybrid retrieval. The archive is cumulative: every compaction adds to it and nothing is cleared, so a later compaction can still find what an earlier one evicted. It lives in its own scope rather than the thread's document scope, because thread documents are injected in full on every request and would re-inject the whole history. Idempotent by content hash, since the same turns are evicted again on every later request. Every entry point degrades to a no-op rather than raising. * Studio: recall archived turns on the turn that evicted them Given only a search tool, a model decides for itself whether to look, and mostly does not. Measured on MRCR v2, a 35B declined on 56% of rows, scoring 0.099 when it skipped against 0.461 when it searched. Forcing one retrieval on the compaction turn took tool-only 0.258 to 0.604, and the model then called the tool on 0% of rows, so the common path costs nothing extra. Recall fires at most once per request, since the tool loop refits on every iteration. The tool loop renders it as an ordinary tool exchange through the builder shared with document auto-inject; the plain path prefixes the latest user message instead, because it sends no tools array and a tool role without one breaks strict chat templates. Only scalar counts join the context_truncated event, so no message content reaches the wire. * Studio: let the model search a compacted conversation Forced recall answers the turn that evicted, but a later turn can refer to something the forced pass had no reason to fetch. Add search_conversation so the model can go looking, scoped to the thread's own archive and sharing the admission slot with document search so the two cannot race for the embedder. The tool is offered only once a thread has actually had turns archived, so an ordinary short chat never pays for the schema, and it is classified read-only so auto mode does not prompt on every call. A matching system-prompt note tells the model the session was compacted, since otherwise it assumes the conversation began where its visible context begins. Deleting a thread now drops its archive rather than leaking a scope per chat. * Studio: read thread_id defensively when selecting tools _select_request_tools also serves the token-count request model, which has no thread_id field, so the archive gate raised there. * Studio: retrieve archived turns lexically first Recalling your own conversation is mostly an exact-match problem: a name, a number, an identifier someone pasted twenty turns ago. Those live or die on rare-token matching, and hybrid fusion was losing them. Measured on a 30-turn walkthrough of a 230k-character document at a 16k window, where every turn shared the same wrapper text. The chunk holding the needle ranked 3rd lexically at any k, was never returned by dense retrieval at all, and RRF pushed it to 16th because it had 30 useless dense hits to fuse with. End to end the model answered with the exact code once lexical leads, and could not answer at all before. Dense still fills whatever the lexical pass leaves, for paraphrased recall. * Studio: keep recall on the branch the user is actually on Editing an earlier message rewinds a thread and continues down a new branch, but the archive is append-only and still holds everything the abandoned continuation produced. Verified against a live build: after rewinding past a turn, querying its distinctive text still returned it, so the model could be handed a turn that on this branch never happened. Recall now drops archived turns that are absent from the thread's saved transcript. Threads with no saved transcript are left unfiltered, since an API caller may pass a thread_id without persisting messages and an empty transcript is absence of evidence rather than evidence the turns are gone. Containment on a normalised prefix rather than a digest, because the archived copy is rendered from the inference projection and the saved copy comes back through the message store. * Studio: show a persistent compaction notice on the turn that compacted The only signal that a long chat had been compacted was a toast, which vanishes after a few seconds and does not survive a reload. A user who scrolls back later has no way to find out why the model seemed to forget the start of the conversation. The notice renders from metadata.custom.contextTruncation inside the assistant message's own container, so it is not part of the conversation sent to the model, is not editable, and is not exported as content, but it stays attached to the turn it describes. It reports how many messages were dropped and, when the conversation archive is on, that they are still searchable and how many passages were recalled. * Studio: show the compaction notice once, on the turn it started A thread that has outgrown its window compacts on every turn from then on, not just the first, so a notice per compacted turn was a notice on every reply for the rest of the conversation. The user needs telling once. The notice is now gated on being the first compacted assistant turn in the thread, found by walking the thread rather than assuming the compacted turns are contiguous or that this is the last one, so a rollback that removes the turn it was on moves it to whichever turn now compacts first. The wording follows: it describes the state the conversation is in from here on, and carries the counts from the turn it began on in parentheses. * Studio: make compaction an occasional event instead of every turn The fit was stateless. The client re-sends the whole saved transcript on every request, so "keep the newest N tokens" recomputes from scratch each time and slides forward a turn or two at a time. Measured on a 40-turn thread against an 8k window, the eviction boundary moved on 12 of 40 turns, which means every few replies quietly lost a little more of the conversation, llama-server's prefix cache was thrown away each time the head of the prompt moved, and there was no such thing as a compaction event to tell the user about. Two changes make it discrete. The fit now reads back the boundary the thread last compacted to, from the newest assistant turn's own persisted truncation, and reapplies it before deciding anything, so nothing new is stored and it survives a restart. And when the boundary does have to move, the trim takes a further ROLLING_COMPACTION_ HEADROOM_RATIO of the budget out (default 0.25) rather than skimming to the brim, so the new boundary has room to stay put. Same 40-turn thread, same window: 4 compactions over 60 turns instead of 14, keeping about 82 percent of the usable budget. Both are gated on the prompt not already fitting, exactly as the recall reserve is. A conversation inside its window is never evicted to satisfy either, and a stale boundary from a branch that was rolled back cannot evict a chat that now fits. The notice follows: it is shown when dropped_messages rises above the last turn that reported it, so it appears once per compaction and stays quiet in between. * Studio: pin that the compaction notice can never become conversation The notice is a sidecar rendered from metadata, not a message, and the ways that could quietly stop being true are all one careless edit away: moving it inside MessagePrimitive.Parts would make it a content part, and everything that walks parts would then replay it to the model, copy it and export it. Asserts it renders as a sibling of the content parts, that neither the outbound message builder nor the assistant replay serialiser reads the key it renders from (bounded to those function bodies, since the streaming handler reads the same key legitimately on the way in), that the markdown export does not mention it, and that it is suppressed while editing so it cannot be typed into the textarea and saved back as text. * Studio: say which part is too long when nothing can make a turn fit If the message just sent is itself bigger than the window, no amount of eviction helps. The fit already handled that correctly, returning the conversation untouched so the request reaches llama-server's normal context-length error, but what the user was then told was actively misleading: the error reports the size of the WHOLE conversation and advises shortening it, when the history has already been evicted and the single message is the part that does not fit. Measured at a 4096-token window: a 5000-token message produces 'Message too long: 10290 tokens ... shorten the conversation', and shortening it cannot possibly work. The fit now returns a fits:false diagnosis instead of a bare None, carrying what the conversation could not be reduced below and how much of that is the latest turn. Every consumer already gated on fits, so this is inert wherever a truncation is treated as a compaction; the streaming paths now forward it so the client can use it. The toast reads the diagnosis and, when the latest turn alone exceeds the window, says so with the numbers instead of offering advice that leads nowhere. The merge drops the diagnosis once a later refit succeeds, by delete rather than by assigning undefined, so an ordinary response keeps exactly the shape it had before. * Studio: fix the review findings on the conversation archive Fifteen items, each reproduced against the code before changing anything. Correctness in the request path. The fit reported a successful recall-capable fit whenever the result was under the prompt budget, but protected messages can stop the trim reaching the reserve target, and the recall then went in anyway: reproduced at ctx 8000, the fit accepted at 6900 and recall took the request to 8948, past the window it had just been made to fit. Recall is now sized from the room the fit actually obtained, and skipped when there is none. Separately, the sticky boundary describes the original transcript, so re-applying it on a later tool-loop fit evicted another boundary-sized block of live history: measured, a second fit dropped 28 of the 30 surviving messages instead of 14, and the summed count persisted an inflated boundary for the next request. It is now spent after the first fit of a request. Availability and privacy. enabled() trusted RAG_AVAILABLE, which only records that import sqlite_vec worked; rag_available() exists because the native vec0 library it loads is a separate file a venv can lack. On such a machine the fit held a recall reserve back, evicting extra history, and then both the archive write and the recall failed, so the user paid for content they never got. And a temporary chat is never written to studio.db, yet the frontend still sends its thread_id and the request carries no incognito flag, so its turns were archived to a scope no deletion flow could reach. Archival now requires the thread to be persisted, which is the same rule that keeps every archive reachable by a delete. Clear-history and project deletion drop archives too; only DELETE /threads did. Archive integrity. The document was committed before its chunks, so a failed chunk write left an empty row marked completed that document_by_hash then skipped forever; both now go in one transaction. The live-branch filter accepted a turn on its first matching line, so editing only the assistant half kept serving the old answer; the whole turn must be present. Retrieval fetched exactly k before that filter, so stale turns could starve live ones and recall returned nothing; it over-fetches first. Tool turns archived only the tool name, which cannot answer what was actually run, so a bounded rendering of the arguments and the assistant text goes in as well. The tool surface. Studio always sends an explicit enabled_tools array and has no reason to name an internal tool, so the allowlist filter removed search_conversation before the archive gate ran and the tool, plus the compaction nudge gated on it, never appeared in a Studio chat. It now follows the archive rather than the allowlist. The forced recall rendered a tool exchange even when the tool was absent from the catalogue, which is the strict-template hazard the plain path avoids; it picks inline in that case, and always inline for the final-answer request, which sends no tools at all. A model-supplied top_k reached a slice as out[:-1] and returned nearly the whole candidate pool, so it is clamped. Both retrieval tools now share the per-turn search cap; only the knowledge-base one was counted. The UI. A fits:false diagnosis is the fitter reporting it could NOT fit, so toasting that older turns were removed was untrue and burned the once-per-thread flag a later real compaction needed. The too-long advice compared the latest turn against the raw context length rather than the prompt budget, so a 3,500-token message in a 4,096-token window was still told to start a new chat, which fails identically. * Studio: fix the follow-up findings on the archive fixes Four items, three of them about last round's own fixes. Archived tool turns had become permanently unrecallable. render_turn now writes 'assistant called X: args' and 'tool result: ...' lines, while assistant-ui persists a tool call as a structured tool-call content part that the transcript flattener dropped; with every archived line required to appear in that transcript, no tool turn could ever match. The transcript now flattens toolName, args and result, and the probe strips the 'assistant called <name>:' label, which is ours rather than the stored message's. The branch probe compared only the first 160 normalized characters of each line, so an edit to the tail of a long answer left the stale copy eligible. Ordinary lines are now compared whole; tool results keep a prefix, since render_turn deliberately truncates those and the archived copy is not meant to equal the stored one. The forced recall sized itself by dividing the remaining budget by CHUNK_TOKENS, which is an embedding-token limit rather than the chat template's cost, and prices none of the wrappers around the injection. It is now recounted with the same tokenizer the fit used and dropped if it overshoots, so the estimate can no longer eat the reply reserve. An omitted top_k on search_conversation defaulted to the clamp ceiling of eight rather than the configured recall default, so an ordinary search could return eight archived turns into the protected current exchange that rolling truncation cannot evict. * Studio: filter conversation recall to the active branch A thread's stored rows are the whole message DAG. Retry and regenerate keep the replaced response as a sibling on purpose, so filtering recall against the whole thread cannot tell a live turn from one the user replaced, and an archived copy of the abandoned response could be recalled into a branch where it never happened. Filter against the messages the request was actually sent with instead, which is one branch by construction, and hand the same branch to search_conversation so the model cannot ask for what the forced recall refused. Falls back to the thread-wide blob for a caller with no branch to offer. Also retunes two respawn-refit fixtures whose windows no longer produced two compactions after compaction started trimming a headroom margin below the budget. * Studio: survive a delete mid-archive, and read the boundary off the active branch Deleting a chat cancels its generation, but cancellation is cooperative and the chunk-and-embed pass between the archive's liveness check and its commit does not observe it. A delete landing in that window drops the thread's rows and sweeps its scope before the commit puts rows back, leaving content the user deleted in a scope no later delete can reach. Re-check after the commit and drop the scope: the delete route removes rows first and sweeps archives last, so either order converges. The sticky compaction boundary had the same thread-wide read as recall did. The stored rows are the whole DAG ordered by creation time, so after a Retry the newest assistant turn can be the sibling the user switched away from, and its boundary is sized for history the active branch does not have. Resolve it against the request's own messages instead. * Studio: give the safetensors loop the same conversation-search guards search_conversation is advertised by thread, not by backend: the tool selector is shared, so a chat compacted under a GGUF model still offers it after the user switches to a safetensors one. The safetensors loop had neither guard the GGUF loop applies to it. It passed no active branch, so a search there fell back to the thread-wide rows and could answer from a branch Retry left behind, and it capped only search_knowledge_base, so paraphrased conversation searches could append archived passages into the protected current exchange on every iteration until the window failed. The shared set of capped retrieval tools now lives beside the cap. The forced recall also derived its query from the loop conversation, which on a later iteration can end with an internal user-role re-prompt rather than anything the user wrote. It reads the request branch's own latest user turn instead. * Studio: widen recall past an abandoned branch, and name whose turn overflowed One over-fetch is not enough for the live-branch filter. Rewinding or retrying a continuation that had already been compacted leaves enough stale turns to fill any fixed candidate window, and the whole page is then rejected while the live match sitting just below it is never examined, so recall reports nothing although the answer is in the archive. Widen and re-ask instead, stopping as soon as there are enough live hits, when the archive stops yielding candidates, or at a bound. The irreducible-fit diagnosis also carried the size of the last message without saying whose it was. A tool loop refits with the tool result appended, so that turn is often output the user never wrote and cannot edit, and the client told them to shorten it. It now reports the role, and the advice splits on it. * Studio: shrink an over-budget recall, and keep truncated tool turns on their branch The exact recount is the right gate, but dropping the whole recall when it fails is the wrong response: with the shipped defaults a full top-K of long turns lands just over the reserve once the wrappers are priced, which would disable the forced retrieval on exactly the long conversations it exists for. Halve the number of turns and re-ask instead, down to one, before giving up. The live-branch probe keyed its prefix rule off the tool-result label, but a long tool result is one appended string containing many newlines, so only its first line carries that label. Continuation lines were compared in full, including the last one, which is the only line the truncation marker is on and can therefore never appear in a transcript: every archived tool turn over the cap was rejected as rolled back. Key off the marker instead, which also covers truncated tool arguments and compares the full text everywhere else. * Studio: retire an edited turn's whole archived copy, not just the edited chunk A turn longer than CHUNK_TOKENS is stored as several chunks of one document, and the live-branch filter ran per chunk. Editing the second half of a long answer therefore retired only the chunks carrying the edit, and an untouched earlier chunk of the same retired turn stayed eligible on its own. The unit that was archived is the turn, which is what the filter already claimed to enforce. Validate every chunk of the candidate's document before admitting any of it, cached per call since candidates from one turn share a document. A query failure falls back to the per-chunk answer rather than failing the recall. * Studio: do not hold a recall reserve back for a chat that is never archived archive_turns refuses a thread with no saved messages, because a temporary chat must not be persisted into a scope no deletion flow can reach. The reserve did not follow that rule: it was granted whenever the RAG stack was available, so an incognito chat, or an API client sending a thread_id without saving anything, paid a full 2,048-token reserve for content that can never arrive. The fit subtracts the reserve from its trim target, so that room is bought with evicted history. Measured on a 4K window: 15 tokens of conversation survived a compaction instead of 1,615. Both now ask one predicate, can_archive, backed by an existence probe rather than a row load. * Studio: match archived turns in order, and delete them without sqlite-vec Independent line membership accepts a turn whose lines were merely rearranged: every probe still occurs somewhere, so the pre-edit ordering stayed eligible and would be served back as what happened. Match the probes in order instead. That only works if both sides agree on the order, and they did not: render_turn writes a tool call before any assistant text on the same message and the result after it, while both transcript builders wrote the text first, so a tool turn carrying both was rejected outright. One flattener now lays out a message the way render_turn does, for the request shape and the stored shape alike, and it offers both JSON spacings for a stored call's arguments, whose object form is not the string the model emitted. Deletion no longer depends on the optional native extension. An archive is only written while vec0 loads, but the library can stop loading afterwards, and a delete that quietly did nothing left a deleted conversation's turns on disk to answer again once it loaded. The fallback removes the text-bearing rows over a metadata connection; the embedding rows it cannot reach carry no text and resolve through tables that are gone. * Studio: give the provider loops the branch, and keep the thread across a respawn The provider tool loops take their catalogue from the same selector as the local ones, so search_conversation is advertised there too once a thread has an archive, but the loop passed no active branch and its searches fell back to the whole stored DAG, where Retry keeps the response it replaced. This is the third loop to need the same wiring. The respawn retry also dropped the thread. It re-enables context_overflow on purpose, to refit an already-compacted prompt for a replacement window, so it is the one path that deliberately compacts a second time: without the thread those extra evictions were archived nowhere, nothing was recalled in their place, and the fit held back no reserve and re-applied no boundary. * Studio: keep the reply that follows a recall, and archive respawn refits group_turns keeps an assistant tool call, its result and the reply that follows in one group, and the archiver rejected any group containing one of our own injections. So on a turn that forced a recall, the model's actual answer was thrown away with it: the question was archived from its own group and the answer was not, and a later search could find what was asked and never what was said. The injections come out now and the rest of the turn stays; retrieved passages are still kept out of the index they came from. The two respawn refits inside the tool loop also evicted without archiving. They run against a smaller replacement window, so those turns are simply gone otherwise. They archive only, deliberately: no reserve is held back on that path, so injecting a recall there is what would push the retry back over the window, and the next request can still recall what this one archived. * Studio: bound the branch check to one turn, and anchor only what recall injected The branch check searched one flattened transcript, so a line an edit removed could be supplied by any later message that happened to repeat the words. Short answers repeat constantly: an archived "Should I deploy? / No" survived its answer being edited to "Yes" because a later turn said "No", and the stale pair stayed recallable. The transcript is now one normalised string per message, and a turn matches only if its lines appear in order within a run of adjacent messages no longer than the turn itself. Recall anchoring took the last two messages of the conversation. That is right for the tool style, which appends a synthetic pair, and wrong for the inline style, which appends nothing and rewrites the latest user message in place: it also pinned the assistant turn before it, and with it an eviction unit the fit was entitled to drop, which can fail a later iteration that would otherwise have fit. The injection now reports exactly what it added or rewrote, and only that is anchored. * Studio: validate a turn's chunks against one run of the branch, not each on its own The chunks of an archived turn are consecutive slices of a single rendering, and each was checked for itself. That let a turn be reassembled out of parts that never sat together: the head matching the question and the answer it has now, the tail matching some later message that happens to repeat the passage an edit removed, so the stale association stayed recallable. Reproduced before the change, with each chunk passing on its own. All chunks of the document must now be found within one run of adjacent messages, bounded by the document's own line count, which is at least the number of messages the turn was rendered from, so a turn that really is still there always fits. * Studio: budget conversation searches and disambiguate identical replies search_conversation clamped the model's top_k only against a fixed ceiling of 8. Eight chunks is roughly 4,000 tokens once wrapped, and the result lands in the current tool exchange, which rolling truncation protects and cannot evict, so on a small context the search itself made the turn unsendable. The GGUF loop now passes the room the window actually has left, the tool clamps top_k by it, and a search with no room says so instead of returning a result that cannot be sent. The sticky compaction boundary took the newest on-branch assistant row, but the branch check is textual, so two siblings whose replies read the same ("Done.") are indistinguishable from there, and Retry is exactly what produces them. Taking the first match applied a boundary measured on a different, deeper branch and evicted live history. Where the text cannot separate them, the smallest boundary is now used: too small costs one more compaction, too large loses turns the branch still has. * Studio: price a conversation search against the whole prompt Three gaps in the budget the previous commit introduced. The GGUF loop subtracted an estimate of the messages alone, leaving the tool catalogue out of the prompt entirely. A large catalogue is thousands of tokens, so the request could already be near its budget while the search was still told there was room for several 500-token chunks. The preflight already prices the request exactly, catalogue and template included, so the difference between that count and the estimate of the same messages is carried forward and the estimate only covers what the loop appends after it. The safetensors loop advertises and executes search_conversation for a thread compacted under a GGUF model, but forwarded no budget, so the clamp in the tool was skipped there. It now passes the room this model has left. With no known context length the argument is omitted rather than sent as zero, which would refuse every search. The sticky boundary filtered candidates with a substring test, which is right for archived chunks and wrong for whole messages: an abandoned "Done" rode in on a live "Not done yet", and having no live twin it then decided the boundary alone. Where any candidate matches a live message exactly, only the exact ones are considered; where none does, the old behaviour stands, since a stored row is not always byte-identical to what the client re-sends. * Studio: keep the overflow diagnosis on the tool path The irreducible-overflow branch counted the newest message on its own, and a tool loop reaches that branch with a tool result last. A tool result by itself is not a conversation: templates that require it to follow its assistant tool call refuse to render one, and the exception escaped the fit entirely, so the caller fell back to the untrimmed request and the client was told nothing at all. That is the one path this diagnosis was added for. The count is now attempted and falls back to the estimator when the template refuses. An approximate number is worth more here than a diagnosis that never arrives. * Studio: budget the conversation search in the provider loop The third tool loop forwarded the active branch but no budget, so the clamp in the tool was skipped there and a model-chosen top_k of 8 could append roughly 4K tokens to a prompt this loop replays on its next call. Studio knows no window for an external model: the request carries no context length and there is no registry to look one up in, and a custom OpenAI-compatible endpoint can be a small local server. So rather than a measured budget, this path spends no more than one ordinary recall's worth, which is the same amount the compaction turn itself is sized for. * Studio: bound an archived turn to its own messages, and archive it once Two problems in the archive. The whole-document branch check bounded the run by the number of LINES the turn produced, and gave each chunk its own start. A turn is two or three messages however long it is, so a hundred-line answer got a hundred-message window, and the tail of an edited answer could be satisfied by a message well outside the turn. The run is now bounded by the messages the turn was rendered from, counted from the labels render_turn writes, and the chunks are scanned as one pass: each continues where the previous one stopped rather than restarting. The cursor inside a message is deliberately not carried over, because chunks overlap and a continuation chunk repeats the tail of the one before it. The hash check that skips an already-archived turn ran long before the insert, with the embedding pass in between, and the index on (scope, sha256) is not unique. Two generations compacting the same thread both cleared it and both wrote, so the turn was stored twice and its copies took two of the few recall slots. The check is now repeated under a write lock immediately before the insert. Reproduced with two concurrent archive passes: two documents before, one after. * Studio: record how many messages an archived turn came from The run an archived turn is allowed to occupy on the branch was bounded by counting the role labels in its rendered text. That counts lines the user wrote as well as the ones the renderer did: a pasted chat log carries lines that look exactly the same, and each one widens the run by a message, which is enough for the message after an edited turn to supply the passage the edit removed. The group's size is now recorded on the document when it is archived, in a nullable column added the way the other lazy upgrades are. Archives written before this have NULL and fall back to the label count, so nothing needs backfilling to keep working. * Studio: persist a compaction boundary the next request can use The boundary was read back from dropped_messages, which counts what each fit removed from the conversation in front of it. The tool loop refits on every iteration and the client sums those counts, so a long agent run added the tool exchanges the turn itself created, and the next request applied the total to its saved transcript. Reproduced with six tool calls on a 4K window: three fits of 4 summed to 12, on a branch that only ever had 4 evictable messages. The boundary is now carried separately, measured against the messages the request was sent with, so it is absolute and re-sending it cannot advance it. The client keeps the latest value rather than summing, and turns saved before this fall back to the dropped count, which is the same number for a turn that fit once. The recall reserve is also dropped once archiving has failed. sqlite-vec can be present and the thread saved while the embedder cannot start: archive_turns swallows that and recall injects nothing, so the room the fit held back was pure loss on every compaction, and the failure mode forgot more history than having the feature off. A failed write marks the archive degraded and the next successful one clears it. * Studio: count the boundary past the system prompt, and notice it Two faults in the boundary added in the previous commit. It stopped at the first message still present, and a Studio request always starts with a system prompt that a fit never evicts, so every compaction recorded a boundary of zero. That is the same as having no boundary: the next request would move the eviction point again and invalidate the prefix cache on every turn. Instruction messages are now skipped rather than treated as the front of the branch, and the newest turn is excluded because it is never evicted and an inline recall rewrites it in place. The compaction notice still keyed off dropped_messages, which is the accumulated count. A tool-heavy turn reporting 12 while the boundary moved to 4 set a high-water mark that silenced the next two real advances. It now reads the boundary, falling back to the dropped count only for turns saved before the boundary was recorded. * Studio: tighten comments in the conversation archive and rolling context window * Studio: give the pasted-text import an extension on this branch too The node test runner cannot resolve an extensionless relative import, so delete-chat-files-preference fails to load the preferences store. main fixed this in d43892ea7; this branch predates it, and the same line is on the branch below it. The change is identical to main's, so it disappears on merge. * Studio: size a conversation search by what it renders to CHUNK_TOKENS is what the chunker aims at, not what a chunk weighs: chunks overlap, the chunker's tokenizer is not the model's, and the rendered block adds markup, source metadata and the tool framing around it. Dividing the budget by CHUNK_TOKENS therefore permitted a result far larger than the room measured, and it lands in the current exchange, which the rolling window cannot evict. Measured on a 500-token budget: one chunk came back at 1,256 estimated tokens. The count is now halved until the rendered result fits, the same backoff the forced recall uses, and a single chunk that still does not fit is refused. The late archive cleanup also spared a recreated thread. DELETE removes the rows, awaits the sandbox pass, and only then sweeps the archive; another tab can POST the same id in that window and its generation can archive turns under it. The sandbox pass re-checks for exactly that, and this now does too, so the recreated chat keeps its memory. * Studio: keep the configured default when top_k is omitted Budgeting an omitted top_k by dividing the whole budget treated the room as a target rather than a cap: on a 128K chat with most of its window free it asked the archive for 200 passages, past the configured default of 4 and past the ceiling of 8 that the model's own value is held to. The default now applies as before, with the room and the ceiling capping it. * Studio: cut the boundary at the newest user turn Excluding only the branch's last message assumed the newest user turn is last, and a continued assistant message puts a prefill after it. The user turn then stayed in the identity scan, and since inline recall rewrites it into a new dict it read as evicted, inflating the boundary by one and costing the next request a live message. The scan now stops at the newest user turn, which is what is protected in any case. * Studio: require an archived turn to end where the live message does Editing a reply by keeping it and adding to it ("No" becoming "No, correction: yes") left every probe matching, so the pre-edit copy stayed eligible and a search could return "No" as the answer with the correction nowhere in it. The scan now reports where it finished inside the message, and the turn is accepted only if it reaches the end of it. Tool results and tool arguments are exempt: render_turn cuts them, so their probes are prefixes by design and demanding the end would retire every turn that carried one. * Studio: check both ends of an archived turn, and charge dense text properly An edit that keeps the old text and adds to it leaves every probe matching whichever side it adds on. The end anchor caught "No" becoming "No, correction: yes"; it did not catch "Correction: no", which ends exactly where the archived copy does. The scan now also reports where the run into the final message began, and the turn is accepted only if it covers that message end to end. render_turn only ever cuts the tail, so the start is required unconditionally. A conversation search sized itself with the shared estimator, which charges four characters per token. That is an English rule: CJK and emoji run closer to one token per character, so a result could be accepted at a quarter of its real size and then land in the current tool exchange, which the window cannot evict. No exact counter is reachable there, the provider loop having no tokenizer at all, so non-ASCII characters are charged one token each and the rest at the usual rate. Measured on 320 CJK characters: 87 tokens claimed, 320 charged. * Studio: earn the compaction headroom, and keep this turn's tool results The headroom cut every overflowing prompt to about 75% of its budget, including on paths that can never put the boundary back: an incognito chat, an API request with no persisted thread, or one whose turns are not saved. There the deeper cut buys nothing and cannot be recalled, so it is simply less history than plain eviction would have kept, and turning the archive off did not restore the old behaviour. The fit now takes the headroom only when the caller can restore the boundary next request. The branch the archive is filtered against was the messages the client sent. A long agent run evicts, and archives, tool exchanges it created itself, and those were then refused as an abandoned branch: the model could not search back a tool result it still needed to answer. The branch is now accumulated across the request, so this turn's own exchanges count as live, while a sibling response from a Retry still does not. * Studio: read the branch as replies when restoring a boundary The rows the rolling window checks are assistant replies, but the branch they were checked against was every message of it, flattened without roles. An abandoned "Done" left by a Retry therefore rode in on a live user message that merely contains it ("not done yet I think"), and with no exact match to prefer, its much larger boundary was applied to a branch that never had that reply. The branch representation now takes a role filter and the boundary asks for assistant messages only. A branch with no reply of its own has no boundary to restore, so it reports none rather than falling back to the unfiltered check. * Studio: account for every message an archived turn claims The anchors covered the final message only, so an edit to the question underneath it left the archived copy eligible, whichever side the text was added on. The scan now checks each message the run touches: it must be matched from the first character, and nothing may be left over when the run moves on. Tool calls are exempt, and the exemption covers the whole message. The store keeps a call as a structured part, so the live text carries the tool name and both spellings of the arguments, spaced and compact, while the archived copy has one line of one of them. Nothing there lines up character for character, and demanding it would retire every tool turn in the archive. * Studio: re-embed archived turns after a model change, and batch the first pass Deduplication was by hash alone, but dense search only reads documents whose recorded embedder matches the query's. A turn archived under the previous model was therefore skipped on every later compaction while being invisible to every paraphrased search, permanently. The check now compares the identity too and replaces the copy, which is what ingestion does with a re-uploaded file. Embedding also ran once per evicted turn. A first compaction of a long chat sent dozens of one-item jobs back to back, and both backends serialise them, so the reply waited for all of them. The turns are chunked first and embedded in one pass, with the per-turn write and its duplicate check unchanged. Measured on 40 turns: 40 embedding calls before, 1 after. * Studio: recall the latest version of a fact, not the most quotable one A long conversation makes its own subject the least discriminative word in its archive. With eight assignments to one variable, the variable name sits in 8 of 17 chunks and scores 0.16 under BM25, while an incidental word from the question ("value") sits in 1 and scores 4.755, so the ranking is decided by the filler and a chunk about a retry budget outranks every chunk that names the variable. Measured over 5 seeds, the newest assignment was archived 8/8 and retrieved 0/5. Three changes, each knob-gated with an off setting that reproduces today byte for byte: - Selection. conversation_match_queries ANDs identifier-like tokens first and falls back to a stopword-stripped OR, used by the archive only, so knowledge base, project and thread search produce identical bytes. Newest revision retrieved 5/5 at both 4 and 8 revisions, zero distractors. - Order. documents carries a nullable archive_ordinal, allocated inside the existing write lock and preserved across the re-embed path, so recalled turns are presented oldest first with a header stating that a later turn supersedes an earlier one. Asking what the value was originally still returns the first assignment, 5/5. - The governing instruction. A standing instruction is protected only while it is the newest user turn, so a short follow-up ("continue") makes it the oldest eviction candidate, and the forced recall then searches the archive for the word "continue". instruction_pin computes a bounded pin through the existing protected_message_ids seam, and a thin latest message adds the last substantive instruction as a second recall query. Pinning ships off; the anchor query ships on, since it changes only what is retrieved. The pin is bounded newest-first over a fixed number of groups, so it protects an instruction against filler rather than against a long conversation. That limit is covered by a test rather than left to be found later. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: stop the focused archive query dropping the answer it was meant to find Six fixes to this branch's own new code, each reproduced before it was written. The focused query could crowd out the very turn it exists to retrieve. The conjunctive identifier pass ran first and broke as soon as it filled `fetch`, so the permissive content-word pass and hybrid search never ran. On an archive of 20 turns all naming one variable, with only the last stating its value, the conjunction returned 16 and saturated the window, and the recall came back with turns 1 to 4 and not the answer, which the pre-change OR query does return. The root cause is ranking, not the threshold: SQLite floors the BM25 IDF of a term appearing in more than half the index at 1e-6 (ext/fts5/fts5_aux.c), so a ubiquitous identifier orders nothing and the cut is effectively arbitrary. The conjunction is now a FILTER over a wide window and the content-word pass supplies the ranking, which is what the two were always meant to do together. Negation is no longer stripped. "What did I say not to delete?" became `say OR delete`, terms a long archive has almost everywhere, and by the same IDF floor they order nothing while the one word carrying the question was gone. `no` and `not` cost nothing where they are common and are the whole query where they are rare. A legacy turn is no longer renumbered when it is re-embedded. Databases upgraded into `archive_ordinal` carry nulls that are deliberately not backfilled, and null sorts FIRST as "older than every numbered turn". Re-embedding after a model change allocated MAX+1 for one of those, so a conversation's oldest turn was rendered as its newest and read as superseding what actually came after it. Merged recall now orders the way single-query recall does. The merge key sorted null turns LAST while `_conversation_order` sorts them first, so a thin follow-up that adds the anchor query produced a block contradicting its own oldest-first header. The key also carries `chunkIndex` now: two queries hitting the same long turn produce equal keys, and a stable sort then quoted chunk 1 ahead of chunk 0. The instruction pin is charged for what it actually holds. The window protects by GROUP and evicts a user-headed group together with its trailing tool groups, so pinning a one-line instruction also holds its reply and its whole tool exchange. Charging only the instruction let a pin exceed its ceiling by any amount: measured at 28 tokens charged against 20037 held, 715 times the cap. An over-budget instruction is now excluded outright, which is what the docstring already promised. One of this branch's own tests was passing on any implementation. `test_re_embedding_a_turn_keeps_its_place` patched the encoder but not `embedding_identity`, so `archive_turns` short-circuited before embedding and returned 0, and its assertion held whatever the code did. That is why the renumbering bug above was not caught here. Hardened to the pattern the neighbouring re-embed test already uses. * Studio: ask the index which chunks are eligible instead of reading it off a capped list Two ways the focused archive query could still drop the answer it exists to find, both of them this branch's own doing and both reproduced before being fixed. Several identifiers are now ORed rather than ANDed. "What are the current values of A123 and B456" is two questions in one envelope, and the turn answering either one names only one of them, so requiring both kept exactly the turns that DISCUSS the pair, which are the older comparisons, and dropped both current assignments. Measured on six comparison turns plus one latest assignment each: the conjunction returned the four oldest comparisons and neither value, while the plain OR path this branch replaced returns both. The filter's job is to keep every slot on something the question asked about, and one identifier out of two is still that. A chunk naming both still outranks a chunk naming one, because it matches more. Eligibility is now asked of the index rather than read off the top of a capped ranked pass. The strict pass is bounded at 256 candidates, and its order inside that bound is arbitrary for exactly the reason this branch already documents: FTS5 floors the BM25 IDF of a term present in more than half the index at 1e-6 (ext/fts5/fts5_aux.c), so the identifier a whole thread is about orders nothing. Past 256 identifier-bearing chunks the turn stating the current value could be absent from that list, get appended behind all 256, and be cut by the fetch window. Measured lost on a 301-chunk archive where the loose pass ranked it first. A new lexical_matching_ids answers membership directly, restricted to the candidates already in hand, which is exact however long the thread gets. It degrades rather than fails: if the probe raises, eligibility is the old set and behaviour is unchanged. Neither existing guarantee was traded away. Both tests that pin the previous behaviour use a single identifier, where the two expressions are identical, and eligibility only ever grows, so identifier-bearing chunks still take every slot. The same edit also removes a latent duplication, where a strict hit appended as fill could appear a second time in the loose tail. Two regression tests, both failing before the fix. * Studio: order merged legacy recalls by when they were said, not by which query found them The merge key matched _conversation_order on the ordinal but dropped its created_at tie-break, and for pre-ordinal rows that tie-break is the whole ordering: every legacy row has turn None, so two of them fell back to chunk index and then to the order the two queries happened to return them, which is the anchor's hits followed by the follow-up's. A later legacy turn could therefore render before an earlier one under a header promising oldest first, on exactly the upgraded databases the null ordinal exists for. The same tie is reachable among numbered rows too, since the ordinal is deliberately not UNIQUE: the write lock is best-effort and two concurrent archive passes can compute the same MAX+1. createdAt is now carried on recall sources beside chunkIndex and the merge key is the same four-part key the single-query path uses. Two tests. One pins the ordering, and one pins the PRODUCER, because nothing renders either field: an unused-looking key is precisely what a later cleanup deletes, and the ordering test alone would keep passing after that. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: stop the archive query stalling on a pasted log, and three narrower fixes The query shaper was quadratic in the length of the question. `_is_identifier` re-scanned the whole question and rebuilt its token list once per distinct token, so a 48 KB paste cost 6,005 full scans and 4.5 seconds, against 1.3 ms for the `_match_query` every other RAG scope uses. At 96 KB it is 17.7 seconds. The reachability is the sharp part: the query is the message the user just sent, and a large paste is exactly what forces a compaction, so the stall lands on the one turn this feature exists for, and again per widening iteration and per rung of the top_k backoff. Purely alphabetic text is the worst case, because a token mixing letters and digits short-circuits earlier and never reaches the scan, so a pasted log or source file is the shape that pays most. Hoisting the tokenization out of the loop is output-identical over 20,000 random inputs and takes 48 KB from 4.56 s to 5.7 ms. The ranking pass is now fetched to the same bound as the filter pass. With the loose window full of content-only distractors, none of them eligible, the merge fell back to the strict pass's order, and that order is worse than arbitrary: on a 160-chunk archive it is a 100-way tie at the 1e-6 IDF floor in rowid order, which is oldest first, systematically the wrong end. The turn stating the current value did not even reach the tie. Loss begins at 20 distractors, 41 turns total. Fetching the ranking pass wider is chunk-for-chunk identical wherever the window was not saturated. A re-embed that stops partway no longer reorders a legacy archive. `create_document` hard-coded a fresh timestamp, so the delete-and-insert re-stamped rows whose null ordinal makes created_at the whole of their ordering. The ordinary embedder switch was already safe, since every compacted request re-presents the whole evicted history and re-embeds it in one pass; the reachable case is a pass interrupted mid-way and served in the same request, which archive_turns is explicitly built to survive. Then the two oldest statements are quoted last, under a header saying later turns supersede earlier ones. A short self-contained request is no longer treated as a nudge. "review billing" and "fix authentication" were classified thin purely for being two words, so an unrelated older instruction was added as an anchor; at top_k 1, which both the recall budget and the over-budget backoff reach, the anchor took the only slot and the user's own words were never searched. Thin now means naming nothing at all rather than being short. One pre-existing assertion flipped with the timestamp fix: a test asserted ordinals [0, None] under ORDER BY created_at, which was incidentally pinning the re-stamp. Its own claim is unchanged. Seven regression tests, each failing before its own fix. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: stop charging an instruction pin for a tool exchange it does not hold * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: stop a tied recall score handing back the oldest revision as the current one FTS5 floors the IDF of a term appearing in more than half the index at 1e-6, which in a per-thread archive is exactly the identifier the conversation is about. Measured on eight revisions of one variable: ONE distinct bm25 across all eight, so the candidate list came back in rowid order, top_k truncated it at the front, and the four OLDEST revisions took every slot. The current value never reached the model. Worse than a miss, because format_conversation_recall states that a later turn supersedes an earlier one, so the stale value was handed over as the authoritative one. Within a run of equal scores only, take from both ends: newest, oldest, next-newest, and so on. Preferring the newest outright is the obvious version and it is wrong, measured: it fails the guard that asks what the variable was set to at the very start, and it fails the new tie test too. Both ends keeps "what is it now" and "what was it originally" answerable out of the same tied run, and nothing moves past a chunk the ranking pass actually separated. Legacy NULL ordinals still sort oldest, and a pass with no ties at all skips the row fetch. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: stop the anchor query shrinking a recall, and make a shouted question still filter Two ways the archive gave back less than it holds. The two-query merge cut each query's slice to its share BEFORE deduplicating, and the anchor is drawn from the same thread as the latest message, so overlap is the normal case: every chunk the two agreed on spent a slot that was never refilled. Measured on six turns matching both queries at top_k 4, either query alone returned 4 sources and the pair returned 2, with four eligible chunks unread. The anchor exists to rescue a thin message and was making the recall smaller than not having it. Over-fetch by what is already held and keep the first `room` fresh rows, sorted by score: the inner call orders its own slice chronologically, so a widened fetch arrives oldest-first and taking its head would put the refilled slots on the oldest turns, which is the tie failure one level up. And the capitals rule in the identifier test needs contrast to mean anything. With no lower case anywhere in the line every word passes it, so the focused pass ORed in "what" and "the" and filtered nothing. Measured: "WHAT IS THE CURRENT VALUE OF ZQXVARA123?" gave '"what" OR "the" OR "current" OR "value" OR "zqxvara123"' where the ordinary spelling gave '"zqxvara123"', and at top_k 1 the one slot went to a turn about a retry budget instead of the variable. Shape still decides, so the identifier is recognised either way; a pure-letter name inside a shouted question falls back to the permissive pass, exactly as the lower-case spelling of the same question already does. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: say on _is_identifier that the capitals rule needs contrast * Studio: keep numbers identifiable in a shouted question, and put the tie-break behind its knob Two follow-ons from the previous commit, both measured. Making the capitals rule need contrast cost purely numeric subjects their shape. "9134" upper-cased is itself, so it qualified only through that rule, and the shape test demanded a letter as well: shouted, the number stopped being an identifier, the focused pass was dropped, and at top_k 1 the slot went to a turn about a retry budget instead of the number asked about. Shape is now "contains a digit", which for any ordinary-case query is the same answer the capitals rule already gave, so nothing else moves. And the tie-break sat outside the rollback knob. RAG_CONVERSATION_QUERY_FOCUS=0 promises the candidate set is identical to before, and reordering candidates is selection, not presentation, so on a tied archive an operator who had turned the feature off still got the new order: measured, the knobs-off recall returned the both-ends set where the previous build returned the four oldest. Gated on that knob, not on the ordering one. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: number archived turns by where they were said, and keep a repeated turn The ordinal was allocated as MAX+1 at archive time, which is only conversation order if eviction is strictly oldest-first, and it is not. truncate_oldest_messages always protects the newest user group, so an agent turn's tool groups are archived while the user message that opened them is still held, and a pinned instruction is archived only once it stops being pinned. Measured: a standing instruction came back as turn 3 of 3, under a header that tells the model the higher number was said later and supersedes the earlier one. The block asserted the reverse of what happened, on shipped defaults. The position now comes from the persisted transcript, grouped with the evictor's own grouper. From the transcript rather than the request because four of the five archive call sites pass the tool loop's already-fitted messages, whose indices shift as earlier groups drop out, while studio.db holds the whole thread and is never truncated. The same read fixes the repeat. Idempotency was keyed on the content hash alone, so a user who says the same thing twice had the second one discarded: "set X to 1", "set X to 2", "set X to 1" stored two documents for three turns, and the turn holding the current value was never indexed, so no query could reach it. A copy now counts as archived only once every occurrence in the transcript has one, and the nth copy takes the nth occurrence's position. Re-archiving an eviction is still free, which is what keeps every later request cheap. Existing archives converge without a migration pass: a turn already indexed under an archive-time number, or under none at all, is re-stamped in place at the next compaction. That is an UPDATE on the row that exists, so nothing is duplicated and no reindex is needed. The re-embed path still keeps its ordinal and its timestamp. Two test expectations move with the contract, both noted where they sit: a repeat that is only a re-eviction now passes persist=False, since appending the turn to the thread a second time describes a user saying it twice, and a legacy NULL row's neighbour is turn 1 rather than 0 because it is the second turn of the conversation. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: match a whole turn to its place, and let an old archive actually converge Two holes in the positional numbering from the previous commit, both measured. A turn was located by its FIRST message, so two different turns that merely start the same claimed each other's seats: a repeated "continue", the same question re-asked, a regenerated reply. Both were stamped with ordinal 0, and since each then believed it had a second occurrence still to fill, the next compaction wrote both again. Measured at four documents for two turns, four recall slots spent on two turns' content, and the older answer quoted under the higher turn number, which the header presents as the one that supersedes. Matched on the whole turn now, as a prefix, because the transcript legitimately lags a turn archived mid-request before its reply is persisted. A turn that matches nothing yields no seats and falls back to the previous allocator, so this can never do worse than not looking. And the re-stamp that was supposed to migrate old archives sat behind the write lock, while the cheap pre-check ahead of the embedding pass skips on exactly the same condition. It was unreachable outside a race, which makes the migration claim in the last commit wrong as it stood: measured, an archive forced back to NULL ordinals still read NULL after a full re-compaction, and one forced into archive-time order 1,0 stayed 1,0 with the recall rendering the second turn first. Both paths now call one helper. The pre-check commits its own update, because that path holds no transaction and the loop returns before the write lock whenever every turn is already archived, which is the ordinary case on re-compaction and therefore the only chance an upgraded archive gets. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: put a persisted tool exchange back into wire shape before reading it The store keeps a tool call as one `tool-call` content PART carrying its own result, while the request carries three messages: the call, the result, the reply. Nothing converted between them, and both readers of the persisted transcript treated the rows as if they were already wire messages, so an agent turn was invisible twice over. `group_turns` splits on `tool_calls`, which a persisted row never has, so the exchange folded into the preceding user group and the evicted tool group found no position of its own. Measured: seats came back empty and the exchange took MAX + 1, the number its own opening question already held, with created_at then ordering the answer ahead of the question under a header saying the later turn supersedes. Worse, `_live_transcript` probes the same rows, so the branch check compared an archived call/result/reply render against a row reading call/reply/result. It matches in order, so it failed: measured, the archived agent turn was filtered out of every recall and no query could return it. Archived content the model can never get back, on every tool-using turn. One reconstruction now feeds both readers. The result moves onto its own `tool` message so each piece renders once, in the order render_turn wrote it, and only the call id goes into `tool_calls` so the arguments still render from the content part in both JSON spellings and `_is_injected` still sees the id it filters our injections by. Seats compare a tool call as needle in haystack for that reason, since the store holds the arguments as an object and the request holds the model's raw string; every other message still compares exactly. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: name the encoding on this branch's test reads, and re-pin the permission read Two CI failures in Repo tests (CPU), both in the test tree. Three read_text() calls added by this branch take the platform default encoding, which the repo's own guard refuses because they break on Windows the moment the file they read gains a non-ASCII byte. Named utf-8, as the guard asks. And the deep-research contract still pinned the literal `const permissionMode = loadPermissionMode();`, which #8686 removed when it put a chat-scoped override in front of that read. The branch already carries the refactored store, so the assertion could not pass. Ported main's replacement rather than deleting the check: the read is still the contract, scoped to the deep-research setter so it cannot be satisfied by the initial-state constant, which is a different property. Inherited rather than introduced here, and it fails the same way on the base branch, but it is what is red on this PR. * Studio: five fixes to the archive's ordering, matching and filtering The refill in the two-query merge sorted by score, which is rounded for display and is identical across a tied run, so it left the list in the chronological order imposed just above and spent its slots on the oldest turns. Measured on eight revisions all scoring 1.6297: the single query returned the newest and the same query plus an anchor did not, so the anchor made the recall worse than not having it. Retrieval rank is now carried on the source and sorted on first. A position SHORTER than the evicted turn could prefix-match anywhere, because zip stops at the shorter side. A thread carrying an orphan user row -- a reply deleted from the thread, or a reload before the reply was appended -- gave the later, answered turn two seats and a second copy at the next compaction, with the recall quoting one turn twice. A short position now matches only the trailing turn. Longer positions stay legal, since _archivable strips injections the transcript keeps and _as_wire expands one tool row into several. list_chat_messages is an unfiltered select over what is really a tree: parent_id is a column and Retry leaves the replaced reply as a sibling. Read flat, that sibling lands between two live turns and the grouper glues it onto the one before it, so the regenerated turn matched nothing and took MAX + 1 -- which the cumulative archive has already pushed past every live turn. Measured: a regenerated turn 2 numbered 5 out of 4 live turns, colliding with live turn 3. Positions now walk the active chain, newest leaf to root, the shape the frontend already uses to decide what the model is shown. A rewind that removes a repeated turn leaves more copies than the conversation has occurrences. Both are byte-identical, so the branch filter passes each against the one surviving occurrence and recall dedups on chunk id, which differs: a slot went on quoting one turn twice, and the surplus kept an ordinal a later turn had since taken. The surplus is retired where it is restamped. And treating any digit-bearing token as an identifier was too wide. The capitals rule it replaced carried a length bar, so only numbers of one or two characters actually changed behaviour, and my previous commit message was wrong to say nothing moved: "answer in 2 sentences" began filtering the archive on "2". Measured at top_k 1 on an archive whose filler mentions small numbers in prose, the focused pass returned the wrong turn where both the previous build and the rollback knob returned the right one. The length bar is back, and a token mixing letters and digits is still a name at any length. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: re-pin the permission read the deep-research contract lost to #8686 The contract still asserted the literal `const permissionMode = loadPermissionMode();`, which #8686 removed when it put a chat-scoped override in front of that read. This branch already carries the refactored store, so Repo tests (CPU) cannot pass here or on anything stacked above it. Ported main's replacement rather than dropping the check. The read is still the contract, and it is scoped to the deep-research setter so it cannot be satisfied by the initial-state constant, which is a different property and would keep passing if the read were deleted. * Studio: retire a surplus copy on the re-embed path too Surplus copies were retired only where a turn was found already archived, and the re-embed path never reaches that branch: it replaces one copy's vectors and returns. So a repeat archived twice and then rewound kept its extra copy as soon as the embedder changed. Measured after a rewind plus an embedder change: three documents for two turns, the recall quoting the repeated turn twice, and the surplus still holding a position a later turn had taken. Retire only, never restamp. Numbering the row here would break the thing the re-embed path is careful about: a turn archived before the ordinal column existed must stay unnumbered, or it moves to the end of its own conversation and the header presents it as the latest word. Measured while writing this, by doing it wrong first: the legacy row came back as 0 and the two guards covering that fired. The surplus rule is one helper now, shared by both callers, and it still does nothing without seats, so a turn that failed to match its occurrences at all never loses a copy. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: re-derive the compaction boundary by position, not by count `boundary_messages` is an absolute count against the transcript it was counted on. Delete an already-evicted prompt from a compacted thread and the front of that transcript gets shorter, but the count is replayed unchanged, so the cut lands that many messages too deep and evicts turns that are still live. Reproduced on `fit_rolling_context` with a 30-turn chat at ctx 2000: a fresh fit keeps from turn 16; deleting one already-evicted pair and replaying the stored count keeps from turn 17, one live turn lost. The loss is then baked in, since the next turn records the boundary against the shortened transcript. Stamp `boundary_anchor`, the text of the first message the fit KEPT, next to the count at all five sites that write it, and look it up on the next request's own branch. The anchor is only ever allowed to make the boundary SHALLOWER, so a stale, ambiguous or repeated anchor costs one extra compaction and can never evict a live turn. Rows written before the key exists behave exactly as today. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: say what the archive kill switch does and does not cover RAG_CONVERSATION_ARCHIVE=0 stops indexing, recall and the recall reserve. It does not turn the rolling window back into what it was before it learned to compact in discrete events: the headroom and the sticky boundary belong to the window, and the headroom has its own knob. "Off restores plain eviction" read as though it covered all of it. Comment only. Gating the window's own behaviour on this flag would make a host with no sqlite-vec silently compact differently from one that has it. * Studio: do not archive a repeat the model can still read Seats come from the persisted transcript, so a turn said twice contributes two of them even when only the older one has crossed the boundary. The same evicted group is handed back on every later request while the sticky boundary holds, so the second pass saw one stored copy against two seats, decided the archive was short, and wrote a document for the occurrence still sitting in the prompt. Measured: two documents for one evicted turn, both recallable, identical text taking two of the four recall slots and one of them repeating what the model can already read verbatim. Bounded by what the fit KEPT. `seats` still allocates ordinals and still drives retire and restamp, unchanged; a separate write budget counts only the seats whose turn is no longer in the prompt. When that turn is evicted later the budget rises on its own and the copy is written then, at its own ordinal. `live` is optional, so every direct caller keeps the previous behaviour. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: charge dense text honestly, and reject an edit inside an archived turn Two fixes. The safetensors search budget priced what is already in the prompt at four characters per token. That is about right for English and roughly half the truth for CJK and emoji: measured on an 81-message Japanese chat, 1295 estimated against 2737 real, reporting 1777 tokens of room where 335 remained. The tool result is already charged a token per non-ASCII character; this is the same rule applied to the spend, which was the missing half. That path runs no rolling fit, so nothing downstream recovers once the exchange lands. The GGUF site takes it only on its fallback leg, since the fit's exact tokenizer count needs no correction, and eviction keeps the flat estimate: making eviction pessimistic would drop history a request could have kept. The probe scan anchored the start and the end of a run but not the middle, so a correction inserted between two archived lines matched both probes with the new line sitting unexamined in the gap, and the pre-edit turn stayed recallable. Verified on "A\nB" becoming "A\ncorrection\nB". A gap is now allowed only where it is a label render_turn wrote and the probe had therefore stripped, since a pasted chat log legitimately carries its own "user:" lines, and the tool-call exemption still covers the rest of its message. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: reach both ends of a tied run, and finish a re-embed that owed a copy Two archive fixes. Every hit on the identifier a per-thread conversation is about ties at the FTS5 IDF floor, and SQLite returns a fully tied run in rowid order, so a candidate cap of 256 means "the oldest 256". Past that many chunks naming the identifier the newest assignment never became a candidate at all, and the ends-first ordering could only reorder the window it was handed. Measured on a 300-turn archive: the window held ordinals 0-255 of 0-299 and the value set by the last turn was unreachable at any k. Both the filter pass and the ranking pass now fetch half from each end, ordered by archive_ordinal rather than rowid, and the merged run is ordered once. Ordinals now come back 295, 0, 294, 1 instead of 255, 0, 254, 1, so "what is it now" and "what was it originally" stay answerable from the same tied run. Rowid was tried first and is not good enough: a re-embed deletes and reinserts a chunk while keeping its ordinal, which scrambled exactly the rows this leg exists to reach. A replacement is not an addition. When the embedder identity changed between a repeated turn's first copy being archived and its later occurrence being evicted, the re-embed branch swapped that copy's vectors and left the count where it was, so the newly evicted occurrence was never written: ordinals [0, 1] where [0, 1, 2] was due, with an intervening contradiction then presented as the conversation's last word on the response the eviction triggered. The copies are now topped up to the write budget, reusing the vectors already in hand. Only where the caller passed `live`, so the budget is a real count of evicted occurrences rather than every seat in the transcript; every direct caller is unchanged. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: end the tool-call anchor exemption where the call ends A stored tool call cannot line up character for character with the live text, because the store keeps arguments as an object and offers both JSON spellings, so the cursor after one is not exact and the character anchors have to relax. That relaxation was then held for the rest of the message. An assistant turn carrying both a call and text therefore stayed matched after a correction was appended to the text: "old answer" becoming "old answer, correction: new answer" left the archived pre-edit turn eligible for recall. Cleared once an ordinary text probe matches, since the cursor is exact again from there. Verified on that exact pair: intact still eligible, edited no longer. * Studio: count live repeats, order both ends by ordinal, restamp a rewound survivor Three follow-ons to the previous archive round. The write budget subtracted live occurrences by set membership, which cannot count. A turn said three times with two evicted and one still in the prompt read as entirely live, so one copy was written where two were due: ordinals [0, 1] instead of [0, 1, 2]. Counted with `_occurrences` now, the same matcher that finds the seats, so the live side is matched by the same rules as the stored one. Only the newest half of a tied run was ordered by archive_ordinal; the oldest half still went by rowid. A re-embed deletes and reinserts a chunk while keeping its ordinal, so the oldest turn's rowid moved from 1 to 297 and it fell out of the front window while the newest-first leg deliberately skipped it. It was in neither half and stopped being recallable at all. Both halves of both passes now take an explicit ordinal tie-break, NULLs first on the oldest side and last on the newest. A re-embed keeps the ordinal it found, which is right while every copy is still there and wrong once one has been retired. Identical turns at ordinals 0 and 2 with the FIRST rewound away left the survivor on 0, so a contradiction at 1 rendered after it and the header, which says the higher turn number was said later and supersedes, handed the model the superseded value. The survivors are restamped against the seats that remain, skipping NULL ordinals so a legacy archive is still never renumbered and `created_at` is still never touched. * Studio: count a video part as media in the rolling preflight `_inject_video_part` writes llama.cpp's own `input_video` part into the same message list the fit is then handed, and the media predicate did not list it. Video prompts therefore ran the rolling preflight, which is skipped for media precisely because `/apply-template` token counting does not include the sampled video tokens. The prompt could be certified as fitting when it does not, or lose history it never needed to lose, and still fail with context_length_exceeded. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: keep which tool was called in the branch probe `render_turn` writes "assistant called <name>: <args>", and the whole label was stripped before probing, name included, so only the arguments decided. A retry that kept the arguments and the result and changed only the tool left the archived pre-edit turn matching: verified on the same arguments moving from terminal to python, which stayed eligible and could be recalled as though the old call had happened on this branch. The name is now its own probe, ahead of the arguments, which is the order `_probe_text` renders a live call in. The label itself still has to go, since it exists only in the archived copy, but the name does not: the live text carries it. `render_turn`'s "tool" fallback for a nameless call is excluded, because the live text has no name there either. Not an exact anchor: the name is matched as a substring of the message, so a tool whose name appears inside the arguments of another can still match. This closes the ordinary retry case rather than every construction of it. * Studio: charge the pin budget for what dense text really costs The pin ceiling exists so a protected instruction stays a bounded minority of the prompt, and it was charged at four characters per token. That undercharges CJK and emoji by roughly 2x or more, so a turn that really costs 1056 tokens was charged 276 and cleared a 1024 ceiling it was nowhere near. The pin then held far more than the budget allows and the fitter evicted recent turns to pay for it, which is the failure the budget exists to prevent. Uses `estimate_messages_tokens_dense`, which is already in the tree for the search budget on the safetensors path. Over-charging here can only ever refuse the pin, never inflate it, which is the safe direction: an unpinned instruction is evicted normally, an over-large pin starves the window. * Studio: do not archive a search the model asked for `_is_injected` recognises the ids this feature and the RAG auto-inject generate, and a search the MODEL issues carries neither: the parser gives it an ordinary `call_N` id. Both the call and the passages it retrieved were therefore indexed as fresh conversation, and a second search archived the first one's output inside its own, one nesting level per distinct search, each copy competing for the four recall slots. Dropped by tool NAME instead, reusing `RAG_SEARCH_TOOLS`, which both loops already share. Only the retrieval parts go: a reply that follows a search is real conversation and is still archived, and an assistant message carrying a retrieval call beside an ordinary one keeps the ordinary one and its result. * Delete the archived conversation even when its thread id comes back The late archive sweep skipped the whole scope when another tab had recreated the id, which spared the recreated chat's memory but kept the deleted conversation's too. The scope is keyed by thread id alone, so those turns stayed recallable in the new chat with nothing left to sweep them, and the endpoint reported success. Take a cutoff before the rows go and bound the delete by created_at when the id has come back: everything archived before the delete was accepted belongs to the deleted conversation, everything after to the new one. Applied at DELETE /threads, project delete and clear all, and on the no-vec0 path so it cannot delete more than the vec0 path does. * Keep pre-call text on the call message, and stop folding case when matching seats Three archive-matching fixes. A persisted assistant row holds the whole turn in generation order, so text before the first tool-call part is what the model said on its way to calling and the live wire form carries it on the call message. _as_wire emitted it after the synthesized results instead, giving the archived copy three messages against the live two, so _occurrences matched nothing and the turn took a fallback ordinal. Split parts at the first call: the preamble rides on the call message, the remainder still follows the results, which is what a reply after the tool result needs. A quoted word is the subject whatever the stopword list thinks of it. 'What did I say about "this"?' reduced to a query for "say" alone, and an archived 'Use this endpoint' was unreachable. Tokens inside a quoted span are now exempt. They stay out of the identifier pass, so this widens the permissive expression only. 'Set key Foo' and 'Set key FOO' hash differently and keep a document each, but lowercasing the transcript comparison handed both seats to both, so the next compaction wrote both again. Seat matching now preserves case; the free-text branch filter, which compares a query against saved text, still folds it. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: tighten rolling context window comments * Number evicted turns on the branch the request is on, not the newest stored row The ancestry walk was seeded from the last saved message, which is not the branch the request is on. Switching to a sibling branch, continuing there and switching back leaves the abandoned branch holding the greatest created_at, and the frontend says as much in refresh-context-usage.ts: after a retry the newest stored leaf is a branch the user left. Measured on a thread with one such switch, positions came back as the abandoned branch's two turns and both of the request branch's evicted turns matched nothing, so each took MAX + 1 over a cumulative archive the other branch had already pushed up. The recall header presents the higher number as the one that supersedes, so an older statement is quoted as the current one. The request branch is already at the call site, so pass it down and seed the walk from the stored leaf whose chain best covers its text. Matching is by text because the wire carries no message ids, and scored rather than compared because the newest branch message is usually not persisted yet. No match and no branch both fall back to the newest row: an empty chain would empty every seat and send every turn to MAX + 1, which is worse than reading the wrong branch. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Stop folding case when validating an archived turn against its branch A turn corrected only in capitalisation still matched its archived copy, so the branch filter kept the pre-edit document live and a recall could answer with the spelling the user had just fixed. Nothing else retires it: the edit changes the digest, so the corrected turn is written as a new document and the branch filter is the only thing that could have dropped the old one. Trim both ends rather than only the right. Keeping leading whitespace looks tighter and is worse, since render_turn strips the whole message and a live turn opening on a space or a newline then starts its run at a non-zero offset and is retired outright. Measured: ' hello there' and a pasted block opening on a newline both went from live to retired. Indentation-only edits stay tolerated in both directions as a result, because probes are matched per line and as substrings; closing that needs the splitter to carry offsets. Also fixes an unrelated retirement found while measuring this. When render_turn's 4000 character cut lands exactly on a newline the marker becomes a line of its own, strips to empty, and was dropped along with the truncation flag, so the last real probe read as complete and an unedited over-cap tool result was retired. Measured on a 900 line result: no query could return it. Note for anyone testing an upgrade in place: boundary anchors written by an earlier build of this branch were stored folded, so they will not match until the next compaction rewrites them. The anchor can only ever move the boundary shallower, so the cost is one extra compaction. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Replay tool rounds the way the serializer does, and stop colliding fallback ordinals Two fixes to how an agent turn is rebuilt and numbered. _as_wire collected every tool-call part of a persisted row into one assistant message and appended every result after it. The replay serializer flushes the pending calls whenever text arrives, so a row holding two sequential rounds goes out as call/result, then the text riding on the second call message, then its result. Rebuilding a different order made group_turns glue exchanges that were separate on the wire, and the later calls matched no position and took an invented ordinal. Parts are now replayed in order with the same flush rule, which also subsumes the earlier special case for text said before the first call. Seats are transcript positions while next_archive_ordinal counts what has been archived, and the newest user group is protected from eviction, so during a tool loop it is in the transcript and not in the archive. A tool group evicted before its assistant row is persisted matched no seat and took the archive's next number, which the user turn then claimed from the transcript. Measured: both documents on ordinal 0, and since created_at breaks the tie the tool answer rendered ahead of the prompt that caused it, under the header saying a higher number was said later. The fallback now takes the transcript length as a floor; a gap in the numbering costs nothing, since ordinals only have to order. * Match the case-preserving probe text in the tool round assertions The archive branch stopped folding case in _normalise, so these assertions, written against the lowercased form, no longer matched. Test-only. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Recognise a unicode ellipsis, fetch both ends for a bare identifier, keep empty tool results Three fixes. Every editor and phone keyboard autocorrects '...' to one ellipsis character, and it survived the punctuation strip, so 'continue...' was called substantive, the anchor was dropped, and recall searched the archive for the word 'continue'. A query made only of an identifier shapes to a single expression, because its focused and permissive spellings coincide, and the single-expression path took the plain one-ended fetch. That is the query most likely to tie on the IDF floor, so it got the oldest rows and the newest assignment was never a candidate. Measured on an archive of cap plus 40 turns: the recall answered with old notes instead of the current value. Only the kill switch takes the plain path now, and one expression returns the filter pass directly rather than running it twice. serializeToolResultPart skips exactly undefined and null and emits a tool message for everything else, a {"result": ""} sentinel for an empty string since the ChatMessage validator rejects empty tool content, and JSON for containers. Treating empty results as absent dropped a message the wire carries, so the reconstructed run was shorter than the archived one and branch validation could filter the turn out of every recall. * Let search_conversation through the Anthropic gate, and probe a tool call before its answer Adding the schema to ALL_TOOLS made the Anthropic selector pick it while _ANTHROPIC_UNPROMPTED_SAFE_TOOLS still listed only web_search and search_knowledge_base, so the pre-switch guard classified a read-only tool as confirmation-gated. Measured: enable_tools with enabled_tools ['search_conversation'] returned 400 on auto and on the omitted default, with the terminal/python message, even though is_potentially_unsafe_tool_call marks it always safe and that set is documented as mirroring it. Pre-PR the same request was served, so this is a regression the PR introduced. Separately, _probe_text bucketed a whole message as call, text, result, which only holds while the text came before the call. A persisted row whose tool call is followed by the model's final answer, the ordinary agent turn, renders as call, result, answer, so _scan_probes advanced past the answer to find the result and could not find it again. Measured end to end on the persisted-row transcript: recall returned the user's question alone and the document holding the answer was filtered out. The buckets now flush when text arrives after a call, the same rule the replay serializer uses; text written before a call still rides ahead of it. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Write the empty tool result sentinel with JavaScript's separators json.dumps puts a space after the colon and JSON.stringify does not, so the reconstructed message read {"result": ""} where the archived one reads {"result":""}. Every comparison downstream is exact, _occurrences with == and _scan_probes with find, and _normalise collapses whitespace runs rather than a single space after a colon, so the message was emitted and the match still failed. That is worse than not emitting it, because it looks fixed. Containers take the same separators, which also closes the pre-existing divergence on a non-empty object. The test now asserts the exact bytes and adds an end to end check that the reconstructed row matches the document archived from the request. * Record the transcript span, and price the recall budget exactly when nothing was trimmed archive_messages bounds the branch check's run, and it was recording the number of messages ARCHIVED rather than the span of the turn. The two differ whenever _archivable drops something: an assistant batch that called search_conversation alongside an ordinary tool archives three messages while the live transcript holds four. Measured, validation failed at three and passed at four, so a perfectly valid ordinary-tool exchange and the answer that followed were rejected as off-branch and could never be recalled. fit_rolling_context returns None when it drops nothing, so a prompt that simply fits, after a context-length increase or on a shorter branch, left the recall budget to a character estimate that cannot see the template's own framing. Measured on a request whose real prompt is about 2800 tokens of a 3584 budget: the budget came back 3512, nearly the whole window, so the recall could append a passage that does not fit and the next iteration cannot evict it again, since the current tool exchange is protected. Priced exactly instead, once per request and only when the model actually reaches for a retrieval tool on a request that did not truncate. A failure falls back to the old estimate. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Replay builtin cards and sandbox wrappers the way the frontend does, and seed the branch in order Three fixes to reconstructing a persisted row. A provider-side builtin is not replayed as an ordinary call. Without a native part the frontend omits it entirely, call and result; with one it replays the call and no tool message, since the result travels in the provider's own part. Reconstructing either as a local call inserted an exchange the request never carried, so the turn matched nothing and took a fallback ordinal. Both signals ride on the persisted args, so the same predicate is decidable here. The name alone never decides, so a user function called web_search is untouched. python and terminal results are wrapped in text, images, sessionId and files on every call, and the replay adapter sends result.text alone rather than feeding the model a session id and file metadata. Serialising the whole wrapper built a tool message that can never equal the archived one. Both of the frontend's gates are mirrored, the shape and the tool name, so a third-party result that merely has text and a session keeps every field it returned. _branch_seed scored leaves by set intersection, which loses repetition and ordering, and leaves are tried newest-first. An abandoned sibling holding the same distinct texts tied and won, so a turn took the seat belonging to its earlier twin and two distinct turns claimed one seat. Scored as an in-order run instead; a multiset would fix the repeat case and not the reordered one. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Price the recall result with the model tokenizer, recount every search, and anchor on tool calls Count the search result with the caller's own token counter instead of the ASCII heuristic, recompute the exact prompt count on every search rather than once per turn, and take the boundary anchor from a message's tool calls when it has no text, so a tool-call message no longer records an empty anchor and disables the rebase. * Mirror the serializer's tool-round flushes and keep the transcript span on copies Drop a tool call the replay serializer drops, flush a completed local pair and a new local round the way it does, and pass the transcript span through to the top-up write so a copy is not bounded by the shorter archived group. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Widen a reused turn's window to its longest span, and price every search exactly A three-message tool exchange and a four-message batch holding the same exchange render identically, so the second is skipped as a duplicate and inherited the shorter window, which the branch check then used to reject it. Grow the stored span instead, upwards only. Count the prompt exactly on every conversation search rather than only when the fit dropped nothing. * Give a tool turn with a preamble its transcript seat The stored probe carries both JSON spellings of the call arguments, and the second one lands between the arguments and any text that followed them, so the live render stops being contiguous inside it. Match the needle in two pieces when it does not fit whole, which tolerates exactly that one insertion. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Let an overlapping chunk restart far enough back to cover its carried messages A chunk's leading overlap is the previous chunk's tail, and it can carry a whole short message with it, so resuming the scan at the message the previous chunk finished in could never match it and retired unedited turns as off-branch. The forward position is still tried first and the walk back stops at the previous chunk's own opening. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Reconstruct a reasoning turn without its thinking, and serialise Unicode like JSON.stringify Reasoning lives in reasoning_content on the wire and never in content, so forwarding the stored part list rendered a reasoning model's thinking where the request sends only the answer. And json.dumps escapes non-ASCII by default while JSON.stringify does not, so a multilingual tool result reconstructed differently from the archived copy. Both cost the turn its transcript seat. * Widen the span on the locked duplicate path as well Both turns can arrive in one compaction: the pre-check clears both before either is written, the shorter is written first, and the longer then met the re-check under the write lock and left the window at the shorter figure. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Tighten comments across the conversation recency changes * Bound the persisted anchor, and spend any room at all on one recall attempt The anchor rides in every truncation event and every assistant turn's metadata while the boundary stays sticky, so a large pasted message was copied across the thread; cap it at a head on both sides, where the read side already clamps only shallower. And a budget below one chunk is not no budget: CHUNK_TOKENS is a ceiling, not the size of a turn, so try one and let the exact recount reject it. * Retire repeats down to the live-aware budget on the already-archived path A rewind or a bigger window can put an evicted occurrence back in the prompt, and re-stamping a copy per transcript seat then left two byte-identical documents, so a recall slot went on text the model could already read. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Break two-ended fetch ties on a legacy archive Every ordinal is NULL there, so both halves ordered by nothing but the score and returned the same rows, which the merge then deduplicated into one end's worth of candidates. Order by created_at in each direction, with the chunk id as a stable last resort. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: alkinun <alkinunl@gmail.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: danielhanchen <unslothshared@gmail.com>	2026-08-19 17:50:25 -07:00
Daniel Han	18b97f8b72	Studio: keep and search the turns rolling context evicts (#9074) ... * Studio: add rolling context windows for local GGUF chat * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: return rolling context metadata for non-stream chats * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: keep original context when rolling fit fails * Studio: keep instruction groups independently protected * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: preserve rolling context metadata across retries * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: count sanitized rolling context prompts * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: refit rolling context after respawn * Studio: scope middle truncation to passthrough * Studio: refit tool prompts after respawn * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: retain later choice truncation metadata * Studio: report clipping-only context truncation * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: expose the turns a rolling fit evicts Rolling context eviction currently drops turns with no way for a caller to learn which ones went. Extract the turn grouping so the eviction unit is reusable, add an identity-based diff of what a fit removed, and let a caller reserve room for content it intends to add back after fitting. The reserve deliberately does not affect whether trimming happens at all, only how far it goes once it is already required, so a conversation that fits today is still returned untouched. * Studio: archive evicted turns into a per-thread rag scope An evicted turn is currently gone for the rest of the session, so the model will state the conversation began wherever its visible context begins. Keep the turns the rolling window drops in a searchable scope built on the existing store, chunker, embedder and hybrid retrieval. The archive is cumulative: every compaction adds to it and nothing is cleared, so a later compaction can still find what an earlier one evicted. It lives in its own scope rather than the thread's document scope, because thread documents are injected in full on every request and would re-inject the whole history. Idempotent by content hash, since the same turns are evicted again on every later request. Every entry point degrades to a no-op rather than raising. * Studio: recall archived turns on the turn that evicted them Given only a search tool, a model decides for itself whether to look, and mostly does not. Measured on MRCR v2, a 35B declined on 56% of rows, scoring 0.099 when it skipped against 0.461 when it searched. Forcing one retrieval on the compaction turn took tool-only 0.258 to 0.604, and the model then called the tool on 0% of rows, so the common path costs nothing extra. Recall fires at most once per request, since the tool loop refits on every iteration. The tool loop renders it as an ordinary tool exchange through the builder shared with document auto-inject; the plain path prefixes the latest user message instead, because it sends no tools array and a tool role without one breaks strict chat templates. Only scalar counts join the context_truncated event, so no message content reaches the wire. * Studio: let the model search a compacted conversation Forced recall answers the turn that evicted, but a later turn can refer to something the forced pass had no reason to fetch. Add search_conversation so the model can go looking, scoped to the thread's own archive and sharing the admission slot with document search so the two cannot race for the embedder. The tool is offered only once a thread has actually had turns archived, so an ordinary short chat never pays for the schema, and it is classified read-only so auto mode does not prompt on every call. A matching system-prompt note tells the model the session was compacted, since otherwise it assumes the conversation began where its visible context begins. Deleting a thread now drops its archive rather than leaking a scope per chat. * Studio: read thread_id defensively when selecting tools _select_request_tools also serves the token-count request model, which has no thread_id field, so the archive gate raised there. * Studio: retrieve archived turns lexically first Recalling your own conversation is mostly an exact-match problem: a name, a number, an identifier someone pasted twenty turns ago. Those live or die on rare-token matching, and hybrid fusion was losing them. Measured on a 30-turn walkthrough of a 230k-character document at a 16k window, where every turn shared the same wrapper text. The chunk holding the needle ranked 3rd lexically at any k, was never returned by dense retrieval at all, and RRF pushed it to 16th because it had 30 useless dense hits to fuse with. End to end the model answered with the exact code once lexical leads, and could not answer at all before. Dense still fills whatever the lexical pass leaves, for paraphrased recall. * Studio: keep recall on the branch the user is actually on Editing an earlier message rewinds a thread and continues down a new branch, but the archive is append-only and still holds everything the abandoned continuation produced. Verified against a live build: after rewinding past a turn, querying its distinctive text still returned it, so the model could be handed a turn that on this branch never happened. Recall now drops archived turns that are absent from the thread's saved transcript. Threads with no saved transcript are left unfiltered, since an API caller may pass a thread_id without persisting messages and an empty transcript is absence of evidence rather than evidence the turns are gone. Containment on a normalised prefix rather than a digest, because the archived copy is rendered from the inference projection and the saved copy comes back through the message store. * Studio: show a persistent compaction notice on the turn that compacted The only signal that a long chat had been compacted was a toast, which vanishes after a few seconds and does not survive a reload. A user who scrolls back later has no way to find out why the model seemed to forget the start of the conversation. The notice renders from metadata.custom.contextTruncation inside the assistant message's own container, so it is not part of the conversation sent to the model, is not editable, and is not exported as content, but it stays attached to the turn it describes. It reports how many messages were dropped and, when the conversation archive is on, that they are still searchable and how many passages were recalled. * Studio: show the compaction notice once, on the turn it started A thread that has outgrown its window compacts on every turn from then on, not just the first, so a notice per compacted turn was a notice on every reply for the rest of the conversation. The user needs telling once. The notice is now gated on being the first compacted assistant turn in the thread, found by walking the thread rather than assuming the compacted turns are contiguous or that this is the last one, so a rollback that removes the turn it was on moves it to whichever turn now compacts first. The wording follows: it describes the state the conversation is in from here on, and carries the counts from the turn it began on in parentheses. * Studio: make compaction an occasional event instead of every turn The fit was stateless. The client re-sends the whole saved transcript on every request, so "keep the newest N tokens" recomputes from scratch each time and slides forward a turn or two at a time. Measured on a 40-turn thread against an 8k window, the eviction boundary moved on 12 of 40 turns, which means every few replies quietly lost a little more of the conversation, llama-server's prefix cache was thrown away each time the head of the prompt moved, and there was no such thing as a compaction event to tell the user about. Two changes make it discrete. The fit now reads back the boundary the thread last compacted to, from the newest assistant turn's own persisted truncation, and reapplies it before deciding anything, so nothing new is stored and it survives a restart. And when the boundary does have to move, the trim takes a further ROLLING_COMPACTION_ HEADROOM_RATIO of the budget out (default 0.25) rather than skimming to the brim, so the new boundary has room to stay put. Same 40-turn thread, same window: 4 compactions over 60 turns instead of 14, keeping about 82 percent of the usable budget. Both are gated on the prompt not already fitting, exactly as the recall reserve is. A conversation inside its window is never evicted to satisfy either, and a stale boundary from a branch that was rolled back cannot evict a chat that now fits. The notice follows: it is shown when dropped_messages rises above the last turn that reported it, so it appears once per compaction and stays quiet in between. * Studio: pin that the compaction notice can never become conversation The notice is a sidecar rendered from metadata, not a message, and the ways that could quietly stop being true are all one careless edit away: moving it inside MessagePrimitive.Parts would make it a content part, and everything that walks parts would then replay it to the model, copy it and export it. Asserts it renders as a sibling of the content parts, that neither the outbound message builder nor the assistant replay serialiser reads the key it renders from (bounded to those function bodies, since the streaming handler reads the same key legitimately on the way in), that the markdown export does not mention it, and that it is suppressed while editing so it cannot be typed into the textarea and saved back as text. * Studio: say which part is too long when nothing can make a turn fit If the message just sent is itself bigger than the window, no amount of eviction helps. The fit already handled that correctly, returning the conversation untouched so the request reaches llama-server's normal context-length error, but what the user was then told was actively misleading: the error reports the size of the WHOLE conversation and advises shortening it, when the history has already been evicted and the single message is the part that does not fit. Measured at a 4096-token window: a 5000-token message produces 'Message too long: 10290 tokens ... shorten the conversation', and shortening it cannot possibly work. The fit now returns a fits:false diagnosis instead of a bare None, carrying what the conversation could not be reduced below and how much of that is the latest turn. Every consumer already gated on fits, so this is inert wherever a truncation is treated as a compaction; the streaming paths now forward it so the client can use it. The toast reads the diagnosis and, when the latest turn alone exceeds the window, says so with the numbers instead of offering advice that leads nowhere. The merge drops the diagnosis once a later refit succeeds, by delete rather than by assigning undefined, so an ordinary response keeps exactly the shape it had before. * Studio: fix the review findings on the conversation archive Fifteen items, each reproduced against the code before changing anything. Correctness in the request path. The fit reported a successful recall-capable fit whenever the result was under the prompt budget, but protected messages can stop the trim reaching the reserve target, and the recall then went in anyway: reproduced at ctx 8000, the fit accepted at 6900 and recall took the request to 8948, past the window it had just been made to fit. Recall is now sized from the room the fit actually obtained, and skipped when there is none. Separately, the sticky boundary describes the original transcript, so re-applying it on a later tool-loop fit evicted another boundary-sized block of live history: measured, a second fit dropped 28 of the 30 surviving messages instead of 14, and the summed count persisted an inflated boundary for the next request. It is now spent after the first fit of a request. Availability and privacy. enabled() trusted RAG_AVAILABLE, which only records that import sqlite_vec worked; rag_available() exists because the native vec0 library it loads is a separate file a venv can lack. On such a machine the fit held a recall reserve back, evicting extra history, and then both the archive write and the recall failed, so the user paid for content they never got. And a temporary chat is never written to studio.db, yet the frontend still sends its thread_id and the request carries no incognito flag, so its turns were archived to a scope no deletion flow could reach. Archival now requires the thread to be persisted, which is the same rule that keeps every archive reachable by a delete. Clear-history and project deletion drop archives too; only DELETE /threads did. Archive integrity. The document was committed before its chunks, so a failed chunk write left an empty row marked completed that document_by_hash then skipped forever; both now go in one transaction. The live-branch filter accepted a turn on its first matching line, so editing only the assistant half kept serving the old answer; the whole turn must be present. Retrieval fetched exactly k before that filter, so stale turns could starve live ones and recall returned nothing; it over-fetches first. Tool turns archived only the tool name, which cannot answer what was actually run, so a bounded rendering of the arguments and the assistant text goes in as well. The tool surface. Studio always sends an explicit enabled_tools array and has no reason to name an internal tool, so the allowlist filter removed search_conversation before the archive gate ran and the tool, plus the compaction nudge gated on it, never appeared in a Studio chat. It now follows the archive rather than the allowlist. The forced recall rendered a tool exchange even when the tool was absent from the catalogue, which is the strict-template hazard the plain path avoids; it picks inline in that case, and always inline for the final-answer request, which sends no tools at all. A model-supplied top_k reached a slice as out[:-1] and returned nearly the whole candidate pool, so it is clamped. Both retrieval tools now share the per-turn search cap; only the knowledge-base one was counted. The UI. A fits:false diagnosis is the fitter reporting it could NOT fit, so toasting that older turns were removed was untrue and burned the once-per-thread flag a later real compaction needed. The too-long advice compared the latest turn against the raw context length rather than the prompt budget, so a 3,500-token message in a 4,096-token window was still told to start a new chat, which fails identically. * Studio: fix the follow-up findings on the archive fixes Four items, three of them about last round's own fixes. Archived tool turns had become permanently unrecallable. render_turn now writes 'assistant called X: args' and 'tool result: ...' lines, while assistant-ui persists a tool call as a structured tool-call content part that the transcript flattener dropped; with every archived line required to appear in that transcript, no tool turn could ever match. The transcript now flattens toolName, args and result, and the probe strips the 'assistant called <name>:' label, which is ours rather than the stored message's. The branch probe compared only the first 160 normalized characters of each line, so an edit to the tail of a long answer left the stale copy eligible. Ordinary lines are now compared whole; tool results keep a prefix, since render_turn deliberately truncates those and the archived copy is not meant to equal the stored one. The forced recall sized itself by dividing the remaining budget by CHUNK_TOKENS, which is an embedding-token limit rather than the chat template's cost, and prices none of the wrappers around the injection. It is now recounted with the same tokenizer the fit used and dropped if it overshoots, so the estimate can no longer eat the reply reserve. An omitted top_k on search_conversation defaulted to the clamp ceiling of eight rather than the configured recall default, so an ordinary search could return eight archived turns into the protected current exchange that rolling truncation cannot evict. * Studio: filter conversation recall to the active branch A thread's stored rows are the whole message DAG. Retry and regenerate keep the replaced response as a sibling on purpose, so filtering recall against the whole thread cannot tell a live turn from one the user replaced, and an archived copy of the abandoned response could be recalled into a branch where it never happened. Filter against the messages the request was actually sent with instead, which is one branch by construction, and hand the same branch to search_conversation so the model cannot ask for what the forced recall refused. Falls back to the thread-wide blob for a caller with no branch to offer. Also retunes two respawn-refit fixtures whose windows no longer produced two compactions after compaction started trimming a headroom margin below the budget. * Studio: survive a delete mid-archive, and read the boundary off the active branch Deleting a chat cancels its generation, but cancellation is cooperative and the chunk-and-embed pass between the archive's liveness check and its commit does not observe it. A delete landing in that window drops the thread's rows and sweeps its scope before the commit puts rows back, leaving content the user deleted in a scope no later delete can reach. Re-check after the commit and drop the scope: the delete route removes rows first and sweeps archives last, so either order converges. The sticky compaction boundary had the same thread-wide read as recall did. The stored rows are the whole DAG ordered by creation time, so after a Retry the newest assistant turn can be the sibling the user switched away from, and its boundary is sized for history the active branch does not have. Resolve it against the request's own messages instead. * Studio: give the safetensors loop the same conversation-search guards search_conversation is advertised by thread, not by backend: the tool selector is shared, so a chat compacted under a GGUF model still offers it after the user switches to a safetensors one. The safetensors loop had neither guard the GGUF loop applies to it. It passed no active branch, so a search there fell back to the thread-wide rows and could answer from a branch Retry left behind, and it capped only search_knowledge_base, so paraphrased conversation searches could append archived passages into the protected current exchange on every iteration until the window failed. The shared set of capped retrieval tools now lives beside the cap. The forced recall also derived its query from the loop conversation, which on a later iteration can end with an internal user-role re-prompt rather than anything the user wrote. It reads the request branch's own latest user turn instead. * Studio: widen recall past an abandoned branch, and name whose turn overflowed One over-fetch is not enough for the live-branch filter. Rewinding or retrying a continuation that had already been compacted leaves enough stale turns to fill any fixed candidate window, and the whole page is then rejected while the live match sitting just below it is never examined, so recall reports nothing although the answer is in the archive. Widen and re-ask instead, stopping as soon as there are enough live hits, when the archive stops yielding candidates, or at a bound. The irreducible-fit diagnosis also carried the size of the last message without saying whose it was. A tool loop refits with the tool result appended, so that turn is often output the user never wrote and cannot edit, and the client told them to shorten it. It now reports the role, and the advice splits on it. * Studio: shrink an over-budget recall, and keep truncated tool turns on their branch The exact recount is the right gate, but dropping the whole recall when it fails is the wrong response: with the shipped defaults a full top-K of long turns lands just over the reserve once the wrappers are priced, which would disable the forced retrieval on exactly the long conversations it exists for. Halve the number of turns and re-ask instead, down to one, before giving up. The live-branch probe keyed its prefix rule off the tool-result label, but a long tool result is one appended string containing many newlines, so only its first line carries that label. Continuation lines were compared in full, including the last one, which is the only line the truncation marker is on and can therefore never appear in a transcript: every archived tool turn over the cap was rejected as rolled back. Key off the marker instead, which also covers truncated tool arguments and compares the full text everywhere else. * Studio: retire an edited turn's whole archived copy, not just the edited chunk A turn longer than CHUNK_TOKENS is stored as several chunks of one document, and the live-branch filter ran per chunk. Editing the second half of a long answer therefore retired only the chunks carrying the edit, and an untouched earlier chunk of the same retired turn stayed eligible on its own. The unit that was archived is the turn, which is what the filter already claimed to enforce. Validate every chunk of the candidate's document before admitting any of it, cached per call since candidates from one turn share a document. A query failure falls back to the per-chunk answer rather than failing the recall. * Studio: do not hold a recall reserve back for a chat that is never archived archive_turns refuses a thread with no saved messages, because a temporary chat must not be persisted into a scope no deletion flow can reach. The reserve did not follow that rule: it was granted whenever the RAG stack was available, so an incognito chat, or an API client sending a thread_id without saving anything, paid a full 2,048-token reserve for content that can never arrive. The fit subtracts the reserve from its trim target, so that room is bought with evicted history. Measured on a 4K window: 15 tokens of conversation survived a compaction instead of 1,615. Both now ask one predicate, can_archive, backed by an existence probe rather than a row load. * Studio: match archived turns in order, and delete them without sqlite-vec Independent line membership accepts a turn whose lines were merely rearranged: every probe still occurs somewhere, so the pre-edit ordering stayed eligible and would be served back as what happened. Match the probes in order instead. That only works if both sides agree on the order, and they did not: render_turn writes a tool call before any assistant text on the same message and the result after it, while both transcript builders wrote the text first, so a tool turn carrying both was rejected outright. One flattener now lays out a message the way render_turn does, for the request shape and the stored shape alike, and it offers both JSON spacings for a stored call's arguments, whose object form is not the string the model emitted. Deletion no longer depends on the optional native extension. An archive is only written while vec0 loads, but the library can stop loading afterwards, and a delete that quietly did nothing left a deleted conversation's turns on disk to answer again once it loaded. The fallback removes the text-bearing rows over a metadata connection; the embedding rows it cannot reach carry no text and resolve through tables that are gone. * Studio: give the provider loops the branch, and keep the thread across a respawn The provider tool loops take their catalogue from the same selector as the local ones, so search_conversation is advertised there too once a thread has an archive, but the loop passed no active branch and its searches fell back to the whole stored DAG, where Retry keeps the response it replaced. This is the third loop to need the same wiring. The respawn retry also dropped the thread. It re-enables context_overflow on purpose, to refit an already-compacted prompt for a replacement window, so it is the one path that deliberately compacts a second time: without the thread those extra evictions were archived nowhere, nothing was recalled in their place, and the fit held back no reserve and re-applied no boundary. * Studio: keep the reply that follows a recall, and archive respawn refits group_turns keeps an assistant tool call, its result and the reply that follows in one group, and the archiver rejected any group containing one of our own injections. So on a turn that forced a recall, the model's actual answer was thrown away with it: the question was archived from its own group and the answer was not, and a later search could find what was asked and never what was said. The injections come out now and the rest of the turn stays; retrieved passages are still kept out of the index they came from. The two respawn refits inside the tool loop also evicted without archiving. They run against a smaller replacement window, so those turns are simply gone otherwise. They archive only, deliberately: no reserve is held back on that path, so injecting a recall there is what would push the retry back over the window, and the next request can still recall what this one archived. * Studio: bound the branch check to one turn, and anchor only what recall injected The branch check searched one flattened transcript, so a line an edit removed could be supplied by any later message that happened to repeat the words. Short answers repeat constantly: an archived "Should I deploy? / No" survived its answer being edited to "Yes" because a later turn said "No", and the stale pair stayed recallable. The transcript is now one normalised string per message, and a turn matches only if its lines appear in order within a run of adjacent messages no longer than the turn itself. Recall anchoring took the last two messages of the conversation. That is right for the tool style, which appends a synthetic pair, and wrong for the inline style, which appends nothing and rewrites the latest user message in place: it also pinned the assistant turn before it, and with it an eviction unit the fit was entitled to drop, which can fail a later iteration that would otherwise have fit. The injection now reports exactly what it added or rewrote, and only that is anchored. * Studio: validate a turn's chunks against one run of the branch, not each on its own The chunks of an archived turn are consecutive slices of a single rendering, and each was checked for itself. That let a turn be reassembled out of parts that never sat together: the head matching the question and the answer it has now, the tail matching some later message that happens to repeat the passage an edit removed, so the stale association stayed recallable. Reproduced before the change, with each chunk passing on its own. All chunks of the document must now be found within one run of adjacent messages, bounded by the document's own line count, which is at least the number of messages the turn was rendered from, so a turn that really is still there always fits. * Studio: budget conversation searches and disambiguate identical replies search_conversation clamped the model's top_k only against a fixed ceiling of 8. Eight chunks is roughly 4,000 tokens once wrapped, and the result lands in the current tool exchange, which rolling truncation protects and cannot evict, so on a small context the search itself made the turn unsendable. The GGUF loop now passes the room the window actually has left, the tool clamps top_k by it, and a search with no room says so instead of returning a result that cannot be sent. The sticky compaction boundary took the newest on-branch assistant row, but the branch check is textual, so two siblings whose replies read the same ("Done.") are indistinguishable from there, and Retry is exactly what produces them. Taking the first match applied a boundary measured on a different, deeper branch and evicted live history. Where the text cannot separate them, the smallest boundary is now used: too small costs one more compaction, too large loses turns the branch still has. * Studio: price a conversation search against the whole prompt Three gaps in the budget the previous commit introduced. The GGUF loop subtracted an estimate of the messages alone, leaving the tool catalogue out of the prompt entirely. A large catalogue is thousands of tokens, so the request could already be near its budget while the search was still told there was room for several 500-token chunks. The preflight already prices the request exactly, catalogue and template included, so the difference between that count and the estimate of the same messages is carried forward and the estimate only covers what the loop appends after it. The safetensors loop advertises and executes search_conversation for a thread compacted under a GGUF model, but forwarded no budget, so the clamp in the tool was skipped there. It now passes the room this model has left. With no known context length the argument is omitted rather than sent as zero, which would refuse every search. The sticky boundary filtered candidates with a substring test, which is right for archived chunks and wrong for whole messages: an abandoned "Done" rode in on a live "Not done yet", and having no live twin it then decided the boundary alone. Where any candidate matches a live message exactly, only the exact ones are considered; where none does, the old behaviour stands, since a stored row is not always byte-identical to what the client re-sends. * Studio: keep the overflow diagnosis on the tool path The irreducible-overflow branch counted the newest message on its own, and a tool loop reaches that branch with a tool result last. A tool result by itself is not a conversation: templates that require it to follow its assistant tool call refuse to render one, and the exception escaped the fit entirely, so the caller fell back to the untrimmed request and the client was told nothing at all. That is the one path this diagnosis was added for. The count is now attempted and falls back to the estimator when the template refuses. An approximate number is worth more here than a diagnosis that never arrives. * Studio: budget the conversation search in the provider loop The third tool loop forwarded the active branch but no budget, so the clamp in the tool was skipped there and a model-chosen top_k of 8 could append roughly 4K tokens to a prompt this loop replays on its next call. Studio knows no window for an external model: the request carries no context length and there is no registry to look one up in, and a custom OpenAI-compatible endpoint can be a small local server. So rather than a measured budget, this path spends no more than one ordinary recall's worth, which is the same amount the compaction turn itself is sized for. * Studio: bound an archived turn to its own messages, and archive it once Two problems in the archive. The whole-document branch check bounded the run by the number of LINES the turn produced, and gave each chunk its own start. A turn is two or three messages however long it is, so a hundred-line answer got a hundred-message window, and the tail of an edited answer could be satisfied by a message well outside the turn. The run is now bounded by the messages the turn was rendered from, counted from the labels render_turn writes, and the chunks are scanned as one pass: each continues where the previous one stopped rather than restarting. The cursor inside a message is deliberately not carried over, because chunks overlap and a continuation chunk repeats the tail of the one before it. The hash check that skips an already-archived turn ran long before the insert, with the embedding pass in between, and the index on (scope, sha256) is not unique. Two generations compacting the same thread both cleared it and both wrote, so the turn was stored twice and its copies took two of the few recall slots. The check is now repeated under a write lock immediately before the insert. Reproduced with two concurrent archive passes: two documents before, one after. * Studio: record how many messages an archived turn came from The run an archived turn is allowed to occupy on the branch was bounded by counting the role labels in its rendered text. That counts lines the user wrote as well as the ones the renderer did: a pasted chat log carries lines that look exactly the same, and each one widens the run by a message, which is enough for the message after an edited turn to supply the passage the edit removed. The group's size is now recorded on the document when it is archived, in a nullable column added the way the other lazy upgrades are. Archives written before this have NULL and fall back to the label count, so nothing needs backfilling to keep working. * Studio: persist a compaction boundary the next request can use The boundary was read back from dropped_messages, which counts what each fit removed from the conversation in front of it. The tool loop refits on every iteration and the client sums those counts, so a long agent run added the tool exchanges the turn itself created, and the next request applied the total to its saved transcript. Reproduced with six tool calls on a 4K window: three fits of 4 summed to 12, on a branch that only ever had 4 evictable messages. The boundary is now carried separately, measured against the messages the request was sent with, so it is absolute and re-sending it cannot advance it. The client keeps the latest value rather than summing, and turns saved before this fall back to the dropped count, which is the same number for a turn that fit once. The recall reserve is also dropped once archiving has failed. sqlite-vec can be present and the thread saved while the embedder cannot start: archive_turns swallows that and recall injects nothing, so the room the fit held back was pure loss on every compaction, and the failure mode forgot more history than having the feature off. A failed write marks the archive degraded and the next successful one clears it. * Studio: count the boundary past the system prompt, and notice it Two faults in the boundary added in the previous commit. It stopped at the first message still present, and a Studio request always starts with a system prompt that a fit never evicts, so every compaction recorded a boundary of zero. That is the same as having no boundary: the next request would move the eviction point again and invalidate the prefix cache on every turn. Instruction messages are now skipped rather than treated as the front of the branch, and the newest turn is excluded because it is never evicted and an inline recall rewrites it in place. The compaction notice still keyed off dropped_messages, which is the accumulated count. A tool-heavy turn reporting 12 while the boundary moved to 4 set a high-water mark that silenced the next two real advances. It now reads the boundary, falling back to the dropped count only for turns saved before the boundary was recorded. * Studio: tighten comments in the conversation archive and rolling context window * Studio: give the pasted-text import an extension on this branch too The node test runner cannot resolve an extensionless relative import, so delete-chat-files-preference fails to load the preferences store. main fixed this in d43892ea7; this branch predates it, and the same line is on the branch below it. The change is identical to main's, so it disappears on merge. * Studio: size a conversation search by what it renders to CHUNK_TOKENS is what the chunker aims at, not what a chunk weighs: chunks overlap, the chunker's tokenizer is not the model's, and the rendered block adds markup, source metadata and the tool framing around it. Dividing the budget by CHUNK_TOKENS therefore permitted a result far larger than the room measured, and it lands in the current exchange, which the rolling window cannot evict. Measured on a 500-token budget: one chunk came back at 1,256 estimated tokens. The count is now halved until the rendered result fits, the same backoff the forced recall uses, and a single chunk that still does not fit is refused. The late archive cleanup also spared a recreated thread. DELETE removes the rows, awaits the sandbox pass, and only then sweeps the archive; another tab can POST the same id in that window and its generation can archive turns under it. The sandbox pass re-checks for exactly that, and this now does too, so the recreated chat keeps its memory. * Studio: keep the configured default when top_k is omitted Budgeting an omitted top_k by dividing the whole budget treated the room as a target rather than a cap: on a 128K chat with most of its window free it asked the archive for 200 passages, past the configured default of 4 and past the ceiling of 8 that the model's own value is held to. The default now applies as before, with the room and the ceiling capping it. * Studio: cut the boundary at the newest user turn Excluding only the branch's last message assumed the newest user turn is last, and a continued assistant message puts a prefill after it. The user turn then stayed in the identity scan, and since inline recall rewrites it into a new dict it read as evicted, inflating the boundary by one and costing the next request a live message. The scan now stops at the newest user turn, which is what is protected in any case. * Studio: require an archived turn to end where the live message does Editing a reply by keeping it and adding to it ("No" becoming "No, correction: yes") left every probe matching, so the pre-edit copy stayed eligible and a search could return "No" as the answer with the correction nowhere in it. The scan now reports where it finished inside the message, and the turn is accepted only if it reaches the end of it. Tool results and tool arguments are exempt: render_turn cuts them, so their probes are prefixes by design and demanding the end would retire every turn that carried one. * Studio: check both ends of an archived turn, and charge dense text properly An edit that keeps the old text and adds to it leaves every probe matching whichever side it adds on. The end anchor caught "No" becoming "No, correction: yes"; it did not catch "Correction: no", which ends exactly where the archived copy does. The scan now also reports where the run into the final message began, and the turn is accepted only if it covers that message end to end. render_turn only ever cuts the tail, so the start is required unconditionally. A conversation search sized itself with the shared estimator, which charges four characters per token. That is an English rule: CJK and emoji run closer to one token per character, so a result could be accepted at a quarter of its real size and then land in the current tool exchange, which the window cannot evict. No exact counter is reachable there, the provider loop having no tokenizer at all, so non-ASCII characters are charged one token each and the rest at the usual rate. Measured on 320 CJK characters: 87 tokens claimed, 320 charged. * Studio: earn the compaction headroom, and keep this turn's tool results The headroom cut every overflowing prompt to about 75% of its budget, including on paths that can never put the boundary back: an incognito chat, an API request with no persisted thread, or one whose turns are not saved. There the deeper cut buys nothing and cannot be recalled, so it is simply less history than plain eviction would have kept, and turning the archive off did not restore the old behaviour. The fit now takes the headroom only when the caller can restore the boundary next request. The branch the archive is filtered against was the messages the client sent. A long agent run evicts, and archives, tool exchanges it created itself, and those were then refused as an abandoned branch: the model could not search back a tool result it still needed to answer. The branch is now accumulated across the request, so this turn's own exchanges count as live, while a sibling response from a Retry still does not. * Studio: read the branch as replies when restoring a boundary The rows the rolling window checks are assistant replies, but the branch they were checked against was every message of it, flattened without roles. An abandoned "Done" left by a Retry therefore rode in on a live user message that merely contains it ("not done yet I think"), and with no exact match to prefer, its much larger boundary was applied to a branch that never had that reply. The branch representation now takes a role filter and the boundary asks for assistant messages only. A branch with no reply of its own has no boundary to restore, so it reports none rather than falling back to the unfiltered check. * Studio: account for every message an archived turn claims The anchors covered the final message only, so an edit to the question underneath it left the archived copy eligible, whichever side the text was added on. The scan now checks each message the run touches: it must be matched from the first character, and nothing may be left over when the run moves on. Tool calls are exempt, and the exemption covers the whole message. The store keeps a call as a structured part, so the live text carries the tool name and both spellings of the arguments, spaced and compact, while the archived copy has one line of one of them. Nothing there lines up character for character, and demanding it would retire every tool turn in the archive. * Studio: re-embed archived turns after a model change, and batch the first pass Deduplication was by hash alone, but dense search only reads documents whose recorded embedder matches the query's. A turn archived under the previous model was therefore skipped on every later compaction while being invisible to every paraphrased search, permanently. The check now compares the identity too and replaces the copy, which is what ingestion does with a re-uploaded file. Embedding also ran once per evicted turn. A first compaction of a long chat sent dozens of one-item jobs back to back, and both backends serialise them, so the reply waited for all of them. The turns are chunked first and embedded in one pass, with the per-turn write and its duplicate check unchanged. Measured on 40 turns: 40 embedding calls before, 1 after. * Studio: re-pin the permission read the deep-research contract lost to #8686 The contract still asserted the literal `const permissionMode = loadPermissionMode();`, which #8686 removed when it put a chat-scoped override in front of that read. This branch already carries the refactored store, so Repo tests (CPU) cannot pass here or on anything stacked above it. Ported main's replacement rather than dropping the check. The read is still the contract, and it is scoped to the deep-research setter so it cannot be satisfied by the initial-state constant, which is a different property and would keep passing if the read were deleted. * Studio: re-derive the compaction boundary by position, not by count `boundary_messages` is an absolute count against the transcript it was counted on. Delete an already-evicted prompt from a compacted thread and the front of that transcript gets shorter, but the count is replayed unchanged, so the cut lands that many messages too deep and evicts turns that are still live. Reproduced on `fit_rolling_context` with a 30-turn chat at ctx 2000: a fresh fit keeps from turn 16; deleting one already-evicted pair and replaying the stored count keeps from turn 17, one live turn lost. The loss is then baked in, since the next turn records the boundary against the shortened transcript. Stamp `boundary_anchor`, the text of the first message the fit KEPT, next to the count at all five sites that write it, and look it up on the next request's own branch. The anchor is only ever allowed to make the boundary SHALLOWER, so a stale, ambiguous or repeated anchor costs one extra compaction and can never evict a live turn. Rows written before the key exists behave exactly as today. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: say what the archive kill switch does and does not cover RAG_CONVERSATION_ARCHIVE=0 stops indexing, recall and the recall reserve. It does not turn the rolling window back into what it was before it learned to compact in discrete events: the headroom and the sticky boundary belong to the window, and the headroom has its own knob. "Off restores plain eviction" read as though it covered all of it. Comment only. Gating the window's own behaviour on this flag would make a host with no sqlite-vec silently compact differently from one that has it. * Studio: charge dense text honestly, and reject an edit inside an archived turn Two fixes. The safetensors search budget priced what is already in the prompt at four characters per token. That is about right for English and roughly half the truth for CJK and emoji: measured on an 81-message Japanese chat, 1295 estimated against 2737 real, reporting 1777 tokens of room where 335 remained. The tool result is already charged a token per non-ASCII character; this is the same rule applied to the spend, which was the missing half. That path runs no rolling fit, so nothing downstream recovers once the exchange lands. The GGUF site takes it only on its fallback leg, since the fit's exact tokenizer count needs no correction, and eviction keeps the flat estimate: making eviction pessimistic would drop history a request could have kept. The probe scan anchored the start and the end of a run but not the middle, so a correction inserted between two archived lines matched both probes with the new line sitting unexamined in the gap, and the pre-edit turn stayed recallable. Verified on "A\nB" becoming "A\ncorrection\nB". A gap is now allowed only where it is a label render_turn wrote and the probe had therefore stripped, since a pasted chat log legitimately carries its own "user:" lines, and the tool-call exemption still covers the rest of its message. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: end the tool-call anchor exemption where the call ends A stored tool call cannot line up character for character with the live text, because the store keeps arguments as an object and offers both JSON spellings, so the cursor after one is not exact and the character anchors have to relax. That relaxation was then held for the rest of the message. An assistant turn carrying both a call and text therefore stayed matched after a correction was appended to the text: "old answer" becoming "old answer, correction: new answer" left the archived pre-edit turn eligible for recall. Cleared once an ordinary text probe matches, since the cursor is exact again from there. Verified on that exact pair: intact still eligible, edited no longer. * Studio: count a video part as media in the rolling preflight `_inject_video_part` writes llama.cpp's own `input_video` part into the same message list the fit is then handed, and the media predicate did not list it. Video prompts therefore ran the rolling preflight, which is skipped for media precisely because `/apply-template` token counting does not include the sampled video tokens. The prompt could be certified as fitting when it does not, or lose history it never needed to lose, and still fail with context_length_exceeded. * Studio: keep which tool was called in the branch probe `render_turn` writes "assistant called <name>: <args>", and the whole label was stripped before probing, name included, so only the arguments decided. A retry that kept the arguments and the result and changed only the tool left the archived pre-edit turn matching: verified on the same arguments moving from terminal to python, which stayed eligible and could be recalled as though the old call had happened on this branch. The name is now its own probe, ahead of the arguments, which is the order `_probe_text` renders a live call in. The label itself still has to go, since it exists only in the archived copy, but the name does not: the live text carries it. `render_turn`'s "tool" fallback for a nameless call is excluded, because the live text has no name there either. Not an exact anchor: the name is matched as a substring of the message, so a tool whose name appears inside the arguments of another can still match. This closes the ordinary retry case rather than every construction of it. * Studio: do not archive a search the model asked for `_is_injected` recognises the ids this feature and the RAG auto-inject generate, and a search the MODEL issues carries neither: the parser gives it an ordinary `call_N` id. Both the call and the passages it retrieved were therefore indexed as fresh conversation, and a second search archived the first one's output inside its own, one nesting level per distinct search, each copy competing for the four recall slots. Dropped by tool NAME instead, reusing `RAG_SEARCH_TOOLS`, which both loops already share. Only the retrieval parts go: a reply that follows a search is real conversation and is still archived, and an assistant message carrying a retrieval call beside an ordinary one keeps the ordinary one and its result. * Delete the archived conversation even when its thread id comes back The late archive sweep skipped the whole scope when another tab had recreated the id, which spared the recreated chat's memory but kept the deleted conversation's too. The scope is keyed by thread id alone, so those turns stayed recallable in the new chat with nothing left to sweep them, and the endpoint reported success. Take a cutoff before the rows go and bound the delete by created_at when the id has come back: everything archived before the delete was accepted belongs to the deleted conversation, everything after to the new one. Applied at DELETE /threads, project delete and clear all, and on the no-vec0 path so it cannot delete more than the vec0 path does. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: tighten rolling context window comments * Stop folding case when validating an archived turn against its branch A turn corrected only in capitalisation still matched its archived copy, so the branch filter kept the pre-edit document live and a recall could answer with the spelling the user had just fixed. Nothing else retires it: the edit changes the digest, so the corrected turn is written as a new document and the branch filter is the only thing that could have dropped the old one. Trim both ends rather than only the right. Keeping leading whitespace looks tighter and is worse, since render_turn strips the whole message and a live turn opening on a space or a newline then starts its run at a non-zero offset and is retired outright. Measured: ' hello there' and a pasted block opening on a newline both went from live to retired. Indentation-only edits stay tolerated in both directions as a result, because probes are matched per line and as substrings; closing that needs the splitter to carry offsets. Also fixes an unrelated retirement found while measuring this. When render_turn's 4000 character cut lands exactly on a newline the marker becomes a line of its own, strips to empty, and was dropped along with the truncation flag, so the last real probe read as complete and an unedited over-cap tool result was retired. Measured on a 900 line result: no query could return it. Note for anyone testing an upgrade in place: boundary anchors written by an earlier build of this branch were stored folded, so they will not match until the next compaction rewrites them. The anchor can only ever move the boundary shallower, so the cost is one extra compaction. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Let search_conversation through the Anthropic gate, and probe a tool call before its answer Adding the schema to ALL_TOOLS made the Anthropic selector pick it while _ANTHROPIC_UNPROMPTED_SAFE_TOOLS still listed only web_search and search_knowledge_base, so the pre-switch guard classified a read-only tool as confirmation-gated. Measured: enable_tools with enabled_tools ['search_conversation'] returned 400 on auto and on the omitted default, with the terminal/python message, even though is_potentially_unsafe_tool_call marks it always safe and that set is documented as mirroring it. Pre-PR the same request was served, so this is a regression the PR introduced. Separately, _probe_text bucketed a whole message as call, text, result, which only holds while the text came before the call. A persisted row whose tool call is followed by the model's final answer, the ordinary agent turn, renders as call, result, answer, so _scan_probes advanced past the answer to find the result and could not find it again. Measured end to end on the persisted-row transcript: recall returned the user's question alone and the document holding the answer was filtered out. The buckets now flush when text arrives after a call, the same rule the replay serializer uses; text written before a call still rides ahead of it. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Record the transcript span, and price the recall budget exactly when nothing was trimmed archive_messages bounds the branch check's run, and it was recording the number of messages ARCHIVED rather than the span of the turn. The two differ whenever _archivable drops something: an assistant batch that called search_conversation alongside an ordinary tool archives three messages while the live transcript holds four. Measured, validation failed at three and passed at four, so a perfectly valid ordinary-tool exchange and the answer that followed were rejected as off-branch and could never be recalled. fit_rolling_context returns None when it drops nothing, so a prompt that simply fits, after a context-length increase or on a shorter branch, left the recall budget to a character estimate that cannot see the template's own framing. Measured on a request whose real prompt is about 2800 tokens of a 3584 budget: the budget came back 3512, nearly the whole window, so the recall could append a passage that does not fit and the next iteration cannot evict it again, since the current tool exchange is protected. Priced exactly instead, once per request and only when the model actually reaches for a retrieval tool on a request that did not truncate. A failure falls back to the old estimate. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Price the recall result with the model tokenizer, recount every search, and anchor on tool calls Count the search result with the caller's own token counter instead of the ASCII heuristic, recompute the exact prompt count on every search rather than once per turn, and take the boundary anchor from a message's tool calls when it has no text, so a tool-call message no longer records an empty anchor and disables the rebase. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Widen a reused turn's window to its longest span, and price every search exactly A three-message tool exchange and a four-message batch holding the same exchange render identically, so the second is skipped as a duplicate and inherited the shorter window, which the branch check then used to reject it. Grow the stored span instead, upwards only. Count the prompt exactly on every conversation search rather than only when the fit dropped nothing. * Let an overlapping chunk restart far enough back to cover its carried messages A chunk's leading overlap is the previous chunk's tail, and it can carry a whole short message with it, so resuming the scan at the message the previous chunk finished in could never match it and retired unedited turns as off-branch. The forward position is still tried first and the walk back stops at the previous chunk's own opening. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Widen the span on the locked duplicate path as well Both turns can arrive in one compaction: the pre-check clears both before either is written, the shorter is written first, and the longer then met the re-check under the write lock and left the window at the shorter figure. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Bound the persisted anchor, and spend any room at all on one recall attempt The anchor rides in every truncation event and every assistant turn's metadata while the boundary stays sticky, so a large pasted message was copied across the thread; cap it at a head on both sides, where the read side already clamps only shallower. And a budget below one chunk is not no budget: CHUNK_TOKENS is a ceiling, not the size of a turn, so try one and let the exact recount reject it. --------- Co-authored-by: alkinun <alkinunl@gmail.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: danielhanchen <unslothshared@gmail.com>	2026-08-19 17:40:23 -07:00
Michael Han	0eb6b6c931	Studio: say when a scan folder cannot be read instead of showing no models (#9053) ... * Studio: say when a scan folder cannot be read A folder Unsloth is denied looks exactly like an empty one: the scan catches the OSError, logs it, and moves on, so the model list is empty with no reason given. Add-time validation now opens the directory instead of trusting os.access, which reads mode bits only and passes on folders macOS TCC or a Windows ACL still refuses. The check runs after the denylist rules so a denied path is never opened. The scan keeps the error it already caught, and both scan-folders endpoints return it as a per-folder status the dialog shows with the setting that fixes it. No new work on the healthy path: 0.06us per folder per scan, and the folder list is a dict lookup with no syscalls. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: record scan folder status from the Hub inventory scan too The folders dialog reads /api/hub/scan-folders, but the scan behind it is the Hub inventory, not collect_local_models, so nothing was ever recorded for it and every row stayed "ok". Its custom-folder loop now records the same way. That alone is not enough: the Hub's _scan_models_dir catches the OSError itself and returns an empty list, so no exception reaches the loop. So an empty result is now the trigger. One opendir says whether the folder is empty, gone, or refused, and it runs only for a folder that returned no models. A folder that found models still costs nothing, with a test that fails if it ever touches the filesystem. * Studio: catch denied model subdirs, and recheck when the dialog reopens Two gaps in the folder status. A root can list fine while every model under it is denied, on a NAS mount or a drive owned by another user. The scanners skip an unreadable child silently, so that arrives as the same empty list as an empty folder and was reported as ok. The probe now also opens subdirectories, stopping at the first refusal and capped at 64, so the denied-everything case costs one extra open. The row tells the user to fix permissions and reopen the dialog, but nothing rechecked between inventory scans, so the warning stayed up after access was restored. Listing the folders now rechecks the folders marked bad, and only those. A healthy folder is not in the registry, so the list still opens nothing. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: probe two levels down, and keep the tests collectable on Windows The child probe descended one level, so <root>/<publisher>/<model> with the model denied still reported ok: both levels above it list fine and the scanners return nothing. It now walks two levels, depth first, on one shared budget of 64 opens. Depth first means a denied mount is found in three opens instead of after every publisher, and the budget bounds the cost whatever the shape of the tree. os.geteuid does not exist on Windows and a skipif condition is evaluated at import, so collecting the test file there raised AttributeError before the os.name check could skip anything. Resolved once into a shared marker, with a test that runs the module body with geteuid removed. * Studio: flag a folder with one denied model, and show status in the picker A folder holding one readable model and one denied model returned the readable one, so the scan looked successful and the denied model was silently absent. The probe now runs whether or not models were found, and reports "partial" in that case so the copy does not contradict the rows on screen by claiming the folder cannot be read. That is a real cost change on the healthy path, so it is measured rather than claimed: 104us for a folder with 8 model dirs, 0.77ms for one with 300, capped by the same 64-open budget. The end-to-end scan stays inside run-to-run variation, and the folder list still opens nothing unless a folder is already marked bad. The old zero-syscall test is replaced by the bound, which is now the guarantee that matters. The inline model selector manages the same folders and rendered only the path, so it shows the status too. * Studio: stop treating an exhausted probe budget as healthy Three fixes. A denied directory past the open budget was reported as ok. Running out of budget means the tail was never looked at, which is not the same as finding it healthy, so it now returns an internal "unknown" that is never recorded and never sent to the UI. That alone would strand a wide folder in a warning it could never clear, so the registry now remembers which directory refused. A recheck opens that one directory, which settles a fixed folder in a single open no matter how wide the folder is or where the denial sat. A folder recorded as partial kept that status after being deleted, because the recheck preserved partial for every non-ok probe. It now only holds partial against a permission result, so missing and unreadable replace it. One permission test was missing the marker that skips it as root and on Windows, where chmod 000 does not deny. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Probe scan folders off the event loop, and stop a vanished model or a Windows device error condemning the folder * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Probe the commit directory inside an HF snapshots folder, and stop calling a shut root partial Two ways the dialog told the user the wrong thing. The cache layout is models--org--name/snapshots/<commit>/, three levels under a registered root, so a probe that stops at two never opens the one directory the weights live in: a denied commit dir made the model vanish from the list while the folder still reported ok, which is the silent empty case this PR exists to remove. The extra level is bought only for a directory named snapshots, since a blanket third level would spend the open budget descending into diffusers component directories. And when the root itself becomes denied, the partial branch restored partial regardless, so a folder none of which can be read kept saying some models in it could not be read, sending the user hunting for one bad model. The cause the probe already returns is the discriminator: it is the root path itself for the root's own refusal and a nested entry path otherwise. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Daniel Han <danielhanchen@gmail.com>	2026-08-19 06:10:58 -07:00
Daniel Han	b0843382d2	Studio: make the chat thread stop getting slower as it fills (#8992) ... * Fix two module resolution failures in the frontend test suite * Fix the third Windows-only path failure in the frontend test suite * Measure how the chat thread's interaction cost grows with message count Studio's chat UI is reported as sluggish on Windows 11 and worsening as the thread fills, while token generation is unaffected. That shape says the cost is per-message renderer work, so the thing to establish first is the curve. New smoke page mounts the real Thread against a synthetic local runtime, seeded to N messages, each carrying prose plus one code fence plus one KaTeX block so Streamdown, Shiki and KaTeX all pay their per-message price. No backend, no auth, no router. New harness runs four scripted actions at N in {10, 50, 200, 500} under 6x CDP CPU throttling: one keystroke into the composer, one scroll gesture, one message action menu opened and closed, and one message delete. Each is bracketed by Performance.getMetrics, which separates the two families of cost: work that grows because layout is uncontained lands in LayoutDuration, work that grows because a listener or an export is O(messages) lands in TaskDuration alone. It measures, it does not gate. There are no timing budgets: it prints the table and exits 0 unless the harness itself broke. Budgets belong in a later change, set from numbers taken on real hardware. What it does fail on is measuring nothing, since that is the failure mode that reads as good news: a seed that did not render, a menu that never opened, a delete that deleted nothing, a scroll that did not move, or four columns that do not rise with N. Measured on this tree, N=10 to N=500: menu open+close 1309ms to 33199ms, delete 438ms to 8491ms, keystroke median 86ms to 268ms. Every metric recorded reaches the printed table, and the harness contract test now enforces that mechanically by parsing the recorded keys out of the source and requiring each one in the table. CDP CPU throttling and longtask are Chromium-only, so Firefox and WebKit runs of this file are correctness checks and not performance ones. * Stop the thread-weight harness charging its own cost to the app A review of the first commit found four ways the measurement could look clean while reporting something other than the app, and two of them were forging part of the curve. All were reproduced before being fixed. The local runtime does have a remote id. It synthesises `__LOCALID_...`, which is truthy, so the per-message fork-count GET fires after all: seeding 20 messages issued 10 requests, against a comment claiming nothing reached the network. They were being answered by a Playwright route handler, so each one paused the renderer for a round trip to another process, once per assistant message. The page now answers them itself, before anything mounts, and the harness fails if a single request escapes during a measured action. Closing the menu was timed from after the Escape dispatch. Radix dismisses synchronously inside it, so the layer teardown, focus restore and re-render -- the O(messages) fan-out this issue is about -- were excluded from the number meant to capture them. Every timing carries a ~33ms floor, since a double rAF cannot resolve faster than two vsync intervals, and CPU throttling does not move it. An action that never happened therefore reported ~33ms, which reads as a plausible measurement rather than as a failure. The floor is now measured per N, printed, subtracted before every growth ratio, and a keystroke at or under it fails the run. The keystroke check read the DOM value back, which is what the harness itself wrote. It now compares against the runtime's own composer state, so a keystroke that reaches the textarea but not React is caught. Also removed from the timed regions: a per-frame document-wide querySelector in both poll loops, replaced by a MutationObserver flag and an isConnected check; a counts() call per frame in the seed gate; an animated scrollIntoView still in flight when the menu window opened; and a console warning per action-bar render, by giving the page the router its useNavigate calls expect. Long tasks are now read after a yield, since the observer delivers on a later task and the tail entry was being dropped. Corrected curve, N=10 to N=500, floors removed where they apply: menu open+close 1021ms to 33591ms, delete 297ms to 8563ms, keystroke 48ms to 283ms, scroll worst frame 5ms to 126ms. Layout stays flat and tiny throughout; the growth is in style recalc and task time. * Stop mounting the assistant action bar for every message At rest the full assistant action bar was mounted under every assistant message. Each one carries around eight tooltips, and every tooltip holds a useSyncExternalStore subscription to the shared modal-layer store, which re-walks its ancestors reading style.pointerEvents whenever Radix puts the body on the modal layer. A 500-message thread therefore mounted 250 bars and 1503 tooltip triggers, and every menu open fanned out across all of them. autohide unmounts rather than hides (ActionBarRoot returns null on the hidden status), so passing it removes the nodes and the subscriptions together. The user bar has always done this. Not unconditionally "always", though: this bar carries the only Stop reading control, which is why it already passes hideWhenRunning={!speaking} and why DeleteMessageButton guards the same case. With "always", moving the pointer off a message being read aloud would take that control away. At most one message speaks at a time, so exempting it costs nothing. Measured with tests/studio/playwright_thread_weight.py at 6x CPU throttle, before -> after, at 500 messages: action bars 250 -> 0 tooltip triggers 1503 -> 3 DOM nodes 56332 -> 41082 delete ms 8595.8 -> 3642.7 scroll worst frame 159.5 -> 87.3 menu open+close ms 33624.5 -> 25279.3 keystroke median ms 316.2 -> 241.0 Note what did not move: menu style recalc, 23725.8 -> 22567.7 ms. A 27% cut in DOM buys 5% there, so the bar is not what makes that number grow. It is the document-wide invalidation from Radix writing pointer-events onto the body, and it is still the dominant cost at large N. The index.css comment is corrected in passing: it justified forcing content-visibility: visible on every code block with "thread length is bounded", which is the assumption this issue disproves. The rule is kept for the flicker it was really fixing. * Studio chat: one fork-count subscription per thread, not one per message The fork badge registered its own CHAT_HISTORY_UPDATED_EVENT listener and issued its own GET, and it is mounted once per message. A delete on a 200-message thread therefore fired 200 requests before anything could repaint, and streaming raises that event once per chunk. Badges now share one debounced subscription per thread and one request that returns every fork count of that thread, so the cost is flat in thread length. * Studio chat: derive research-message ownership once per thread revision useOwnsResearchMessage exported the whole thread from inside a per-message render body, so one render pass over N messages exported N times and inspected N*N items. Streaming re-renders the thread once per chunk, so that pass is hot. The answer is a property of the thread revision, so derive it once for the message list every message in the pass already shares. Measured on a synthetic thread: 200 exports and 0.80ms per pass becomes 1 export and 0.014ms; at 1000 messages 16.4ms becomes 0.02ms. * Studio chat: stop deep-cloning the thread on every delete and every save exportedItemToRecord ran JSON.parse(JSON.stringify(...)) over every message's content and attachments on its way to a PUT that serializes the same records again, and syncExportedRepositoryToBackend ensured the thread row that syncStoredChatMessages already ensures, so every save paid for GET /threads/{id} twice. The parts are replaced rather than mutated, so a copy of the list is snapshot enough and the bytes on the wire are identical (asserted in the new test). Measured on a 200-message thread of ~4KB messages: the record step drops from 1.17ms to 0.015ms, and a delete makes one thread-row read instead of two. * Studio chat: open the message action menu non-modally A modal Radix menu writes pointer-events:none on <body>. That is an inherited property, so every open and close invalidates style for the whole document, and on a long thread the recalc is the bulk of the cost. Non-modal never writes it. Also teaches the harness the difference between a cost that was removed and a page that never mounted one, so the after-tree does not read as broken. * Pin the message action menu to the non-modal layer * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Drop the #8980 content this branch no longer needs * Print the hovered trigger count and keep the modal layer load-bearing in the verdict The contract test caught both: a metric recorded but not printed, and the verdict no longer reading body_pointer_events_while_open after the non-modal fix removed the old check. The layer is now compared ACROSS N instead, since either mode is legitimate but mixing them means the columns measure different mechanisms. * Stop the rapid-submit settle wait measuring the action bar instead of the reply This branch autohides the assistant action bar, and that turned the last wait of the rapid-submit step into a 7-of-7 failure on Windows CI. The wait is not what the step proves, and the clause that broke was not measuring what it claimed. innerText of a [data-role=assistant] root spans the whole subtree, and the action bar sits inside it, so 'every reply has non-empty innerText' was satisfied by button labels regardless of what the model returned. Instrumented at that point on the CI runners, two runs on this branch's merge base read: content=[0, 0] innerText=[73, 73] clause held, BOTH replies empty content=[0, 19] innerText=[73, 89] clause held, first reply empty gemma-3-270m-it answers 'Reply with exactly: rapid-first' with an empty completion in 3 of 8 sampled runs, on the merge base as much as here, and the clause held every time. So the empty reply is the model, is pre-existing, and was simply masked. With the bar autohidden the subtree is content only, and the same empty completion now fails. Dropped rather than repointed at the content element: an empty completion is the model's behaviour, so a content assertion would be flakier than what it replaces. What is left is exactly the settle this wait is for, two bubbles, nothing streaming, nothing queued. The behaviour the step exists to prove, that a 100 ms follow-up queues behind a held first turn, is state.queueSeen above and is untouched. Test-only. No Studio code changes, so the menu open+close and nodes-at-rest wins are unaffected. Verified on Windows CI at this branch's head: 4 of 4 Chat UI Tests jobs green with this change, against 7 of 7 red without it. * Validate the delete measurement at every size, not just the last * Keep the thread's fork counts across the autohidden badges * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Exempt the thread-weight harness from the CI-coverage check * Keep the newest reply's action bar in the tab order * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Load the crypto polyfill on the thread-weight smoke page The page this branch adds was the only one in studio/frontend without <script src="/crypto-boot.js"></script> in its head, so crypto-uuid-boot.test.ts fails with "smoke-thread-weight.html must load /crypto-boot.js". The rule is not cosmetic: the polyfill has to run before the module entry or crypto.randomUUID is missing on the older WebViews Desktop embeds, and a smoke page without it measures a page that differs from production in the one respect the harness is meant to hold constant. Reproduced on this branch and verified: the named assertion fails before the change and the file's four tests pass after it. * Debounce the fork count refresh instead of throttling it onHistoryUpdated returned while a timer existed, which is a leading edge throttle rather than a debounce. Streaming raises the history event once per chunk, so the timer expired mid stream and the next chunk armed another one, costing a whole thread fork count fetch every 300ms for as long as the reply ran. Fork counts cannot change during generation, so all of those were waste. Clear and reschedule on every event, as the sidebar refresh already does. The existing burst test fired all 20 events inside one window, where a throttle and a debounce behave identically, which is why this survived. The new case spreads the events across the window like a real stream: 20 chunks cost 10 refetches before this change and none after, with one refresh in the quiet window that follows. * Reveal the action bar on focus, not only on hover Unmounting the bar on every message but the newest took its controls out of the tab order, and there was no non-pointer way to bring them back, so Copy, Edit, Refresh, Delete, Read aloud and More were unreachable by keyboard or screen reader on every older reply. Deferred rendering is fine while the user can still ask for what is deferred, and tabbing is asking. Measured on the thread weight smoke page at 20 messages, older reply: the accessibility tree exposed no bar controls before this change and still none after focus entered the message; it now gains Copy, Refresh, Delete, Edit response, Read aloud and More. Three tabs from an older reply used to walk its two code fence buttons and step to the next message without ever entering a bar; the same walk now lands on the bar's Copy. The library has no focus path, so this drives its own isHovering flag from focus within the message root. Two writers share that flag, so a pointer leaving while focus is inside re-asserts it from a listener registered in an effect, which runs after the primitive's own mouseleave in the same dispatch and so never renders the intermediate false that would unmount the element holding focus. The clear is a one frame watchdog reading activeElement rather than a relatedTarget test, because relatedTarget is null both for browser chrome and for a portal, and no focusout fires at all when the focused element is removed, which is how the menu closes. At rest this stays at one mounted bar of ten, and at one again after a focus round trip, so the weight this branch is here to remove is unchanged: 1950 DOM nodes and 9 tooltip triggers, the same as before. * Give a plain prose reply a way into the tab order The focus reveal only fires once focus is inside the message, and a reply whose body is plain prose contains nothing focusable after autohide unmounts its action bar. The earlier measurement of two focusable controls per message was an artefact of a fixture where every reply carried a code fence, and Streamdown ships one Copy button per fence. Seeded with a prose only reply it is zero, and Tab walks straight past the message into the next reply, so Copy, Edit, Refresh, Delete, Read aloud and More stay unreachable. tabIndex on the message root rather than a visually hidden button: it adds no DOM node, which matters for a branch that exists to cut per message weight, and it draws nothing at rest. The app's own focus-visible rule gives it the same 1px keyboard indicator every other focusable container already has, and focus-visible means a mouse click still draws nothing. Measured at rest, unchanged: one action bar of ten, 1950 DOM nodes, 9 tooltip triggers. Outline is none with neither focus nor hover, and none after a click. The cost is one extra tab stop per assistant message, which is the price of the controls being reachable at all. The fixture gains an opt in plain prose variant so the existing weight measurements keep the exact thread they had. * Bound how long a fork change can wait behind an unrelated stream CHAT_HISTORY_UPDATED_EVENT fires once per streaming chunk, and the fork-count refresh was a pure trailing-edge debounce, so a reply running in a background thread reset the timer on every chunk. Deleting a fork from the sidebar while looking at its parent changes the displayed count, and the refresh was postponed until the unrelated stream went quiet, which on a long or queued run is minutes. That is starvation, not slowness. The event is a bare Event with no detail and six other consumers, so telling fork changes apart from chunks means changing a contract well outside this store. FORK_COUNT_REFRESH_MAX_WAIT_MS bounds the wait inside it instead: a second timer, started by the first event of a burst and deliberately not restarted by the ones after it, races the debounce, and whichever fires first cancels the other. A second timer rather than a Date.now() deadline so it runs off the same clock as the debounce and is testable without a fake Date. 2000ms because the bound costs one whole-thread fetch per window while a stream runs. At the 300ms debounce that is the per-chunk traffic this store exists to remove; at 2000 it is under a sixth of it, and only while something is streaming. The existing continuous-stream test asserted ZERO mid-stream fetches, which is the behaviour the review flagged, so it now measures the price of the bound instead of claiming there is none: it pins the count against both what the ceiling allows and what a leading-edge throttle would have cost, so a regression in either direction is a failure. Four assertions were made to fail on their own broken tree before being kept: the ceiling removed, the ceiling restarted per chunk so it never expires, the losing timer left uncancelled when the other fires, and the ceiling left running past unsubscribe. That last one was vacuous at first, since the entries map is empty after unsubscribe and a leaked timer refreshes nothing; it now subscribes a second thread inside the ceiling window, which is both observable and the case that actually costs a user a request. * Scope the popup lookup to the action bar The watchdog treated any expanded descendant as this message's open menu. Reasoning cards and tool-fallback cards are Radix CollapsibleTriggers and render aria-expanded=true for as long as the reader leaves them open, which is the resting state of a message whose tool output has been expanded. decide() therefore found a popup every frame, rescheduled itself every frame, held focusWithinRef and the synthetic hover set, and left the bar mounted indefinitely, at the cost of a DOM query per frame per such message. Scoped to .aui-assistant-action-bar-root, which is where the trigger the hook has to hand focus back to actually lives. Proving this took three attempts and the first two were wrong, which is worth recording because the failure was in the test rather than in the fix. isHovering has two writers. This hook is one; assistant-ui's own MessagePrimitive.Root mouseleave handler is the other, and it writes false directly. So a phase that reveals the bar by HOVERING and then moves the pointer away sees the bar unmount on both trees, because the library unmounted it. On the broken tree the watchdog genuinely spins forever, and the bar still goes away. An assertion on the mounted bar count cannot attribute that outcome to this branch, and C2 passed on the fixed and broken trees alike. The phase now keeps the pointer off the message entirely and reveals the bar by focus, which the tabIndex on the message root makes possible. With no mouseleave to fire, focus is the only writer and the bar's fate is decided by the watchdog alone. C2 is green on the fixed tree and red under --break widepopup, 1 bar still mounted and held indefinitely. Two guards sit in front of it: one asserts the pointer really is off the message, the other that the bar really was mounted and focused, so the phase fails loudly rather than passing vacuously if either precondition breaks. Also fixes a pre-existing crash the sweep was hiding. Under --break eagerclear the bar is gone by the time A8 runs, and a bare more.focus() on an undefined element threw a Playwright TypeError that killed the process before phases P, B and C ever ran, so the break reported fewer reds than it earns and ended in a traceback rather than a red result. It is more?.focus() now and the trigger's absence is folded into A8's condition, so eagerclear completes and A8 goes red on its own merits. Full sweep: head 14/23, notabindex 19/23, restring 21/23, focusring 22/23, leakflag 21/23, noreassert 22/23, eagerclear 21/23, widepopup 22/23 with C2 the only red. thread.tsx checksummed before and after all eight runs, identical every time. * Reveal the action bar from the backward traversal too The tabIndex on the message root only worked going FORWARD. A container is reached before its own descendants, so Shift+Tab arriving from the message below landed on the last tabbable thing in the message, and with the bar unmounted that is the root, which sits BEFORE the bar in DOM order. Focusing it mounted the controls and the next Shift+Tab then stepped straight past them to the previous message. Copy, Edit, Delete and More were reachable going forward and unreachable going backward, which is worse than being unreachable outright, because the forward pass makes it look solved. A sentinel span after the bar is what makes the backward pass land inside the message: focus stops there, the bar mounts, and the next Shift+Tab goes into the last control rather than out. It is deliberately NOT a focus redirect to that control, which would trap the forward pass in a loop between the last button and the sentinel. It carries no onFocus of its own because React's onFocus is focusin and already bubbles to the root, and no role, because it performs no action; the aria-label is what stops it being an unannounced stop. It DOES cost one DOM node per assistant message, and this branch is about per-message weight, so that is asserted rather than absorbed: the at-rest guard now requires exactly 1950 baseline nodes plus one sentinel per reply and nothing else, measured 1960 for 10 replies, with the sentinel count checked separately so the extra nodes are attributed rather than tolerated. Three assertions, each proven red. nosentinel takes D2 and D3 red while D1 stays green, which is precisely the reported asymmetry: focus still enters the reply, it just skips the bar. D1 says something weaker, so it needs noentry, which removes the sentinel and the tabIndex together and leaves nothing in the message reachable at all; D1, D2 and D3 all go red there. Full sweep, 26 assertions: fixed 26/26, nosentinel 23/26, noentry 18/26, notabindex 24/26, head 15/26, widepopup 25/26 with C2 alone, eagerclear 21/26, noreassert 25/26, leakflag 24/26, restring 24/26, focusring 25/26. thread.tsx checksummed before and after every run, identical throughout. * Draw a focus indicator on the backward reveal sentinel The shared soft-outline rule is :where(div, main, section, aside, ul, ol):focus-visible, which never matched a span, so the sentinel was a real tab stop that drew nothing: Shift+Tab into a message made focus visibly disappear for one stop before the next press reached the action bar. That is a focus-visible failure, not a cosmetic one. The element stays 0x0 and the ring is drawn by outline-offset. Outlines take no part in layout, so the indicator appears without shifting the message, which giving the span dimensions on focus would have done. Still nothing at rest, and :focus-visible means a mouse click draws nothing either. Measured: with keyboard focus the ring spans 14px against the UA default's 2px on a zero-sized span, which is the difference between an indicator and no indicator. D4 asserts the SPAN of the drawn ring rather than merely that an outline style exists, because the broken tree still reports outline-style auto and would satisfy a presence check while showing nothing. Proven red by --break blindsentinel, which removes the indicator rule and leaves the tab stop itself intact, so only D4 fails. * Stop tracking the generated Studio test database .studio-test-root/studio.db is written at test time by tests/studio/install/test_selection_logic.py, which points storage_roots.studio_root at that path. It is a mutable SQLite runtime database, not a fixture: nothing reads it, any test run or Studio start rewrites it and dirties the checkout, a later accidental commit could capture real local chat or settings data, and it puts 221 KB into every clone while exercising nothing. It was not in the tree deliberately. It arrived in the merge commit here because that commit was staged with `git add -A` after running the suite, and no gitignore rule covered the path. Main does not track it. Untracked and ignored, and the file is left on disk since creating it is normal. --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: danielhanchen <unslothshared@gmail.com> Co-authored-by: oobabooga <112222186+oobabooga@users.noreply.github.com> Co-authored-by: danielhanchen <danielhanchen@users.noreply.github.com>	2026-08-18 06:19:05 -07:00
Maheswar Kumar	5bb0bc6f73	studio: keep each chat's composer pills and settings with the chat (#8686) ... * studio: keep each chat's composer pills and settings with the chat The composer pills, the permission level and the retrieval controls were installation-wide, so switching chats carried one conversation's modes into the next, and reopening an old chat showed whatever the defaults happened to be. Sixteen of those settings now travel with the thread. chat_threads gains a settings_json column, ChatThreadSettings pins the contract PATCH /api/chat/threads/{id} accepts, and thread-scoped-settings.ts re-validates every value against the same literals and ranges before it is sent. Editing one of them with a chat open writes the snapshot onto that thread and leaves the installation defaults alone. Editing with no chat open still moves those defaults, which every chat without a snapshot follows, so a fresh install behaves exactly as before. A chat that stored nothing is pinned on first open, so a later change to the defaults cannot rewrite its modes. Full access stays session-only: the sanitizer drops it, and a write made while it is active carries through the level the chat already held. resolveToolsEnabledOnLoad, setBypassPermissions, setDeepResearchEnabled and both external-provider effects in chat-page.tsx read the open chat's value before the installation one, so a model load or a model switch no longer re-applies the defaults over the pills a chat is running with. A thread that stores no value for a setting falls back to the defaults rather than to the outgoing chat's. upsert_chat_thread COALESCEs the column, so the writers that rebuild a thread record cannot clear it; fork_chat_thread copies it; and list_chat_threads leaves it out, since the sidebar lists every thread and only an opened one reads it. normalizeStoredPermissionMode moves the legacy confirm-toggle migration out of the store so it can be pinned directly. Once the level is mirrored to /api/chat/settings the first hydration seeds it there, so the mapping can no longer be driven through the UI more than once per installation. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Hold a chat's edits until its snapshot lands, and keep deep research on a switch Two fixes on top of the thread-scoped settings work. A pill clicked between opening a saved chat and its GET /api/chat/threads/{id} returning was written to the installation defaults, because captureThreadScopedEdit needs activeThreadId to equal threadScopedSettingsThreadId and the pairing is only set when that read resolves. The click moved every snapshot-less chat's default and was then overwritten by the arriving snapshot, so it leaked and appeared to do nothing. Those edits are now held for the duration of the read: if the chat turns out to own a row they win over what came back and are stored on it, and if it does not (a new chat's runtime id, a legacy row, a failed read) they are replayed to the defaults exactly as before. On loopback the read wins the race, so this shows up on remote and tunnelled sessions. buildThreadScopedSnapshot carried the stored permissionMode forward when apply() held it back under Full access, but not deepResearchEnabled, which apply() holds back the same way for external models and incognito. Toggling any other pill in such a chat erased the true it had stored. Carried forward the same way. * Read a stored snapshot leniently, and keep the beacon for terminal events only Two problems found while simulating upgrades, downgrades and the three browser engines against this branch. settings_json is the first strictly validated nested model Studio builds out of the database rather than off the wire, and a stored snapshot outlives the build that wrote it. A newer Studio adding a setting, widening an enum or raising a bound writes a blob this one rejects, and refusing it 500s the chat on open, on fork and on patch, and takes GET /api/chat/export down for every chat, not just the affected one. _json_loads already shrugs off JSON that will not parse, so JSON that parses but postdates this build now gets the same treatment: rows go through thread_from_row, which drops what it cannot read. Only the read is forgiving. Both wire models stay extra = forbid, so a client still cannot invent a setting, and the row is left untouched, so upgrading again restores it. The unload flush also ran on visibilitychange, which is not terminal: it fires on every tab switch and the page carries on afterwards. That path used the keepalive beacon, which PATCHes the row directly and answers 404 for a thread whose row has not been created yet, while having already consumed the pending write. Terminal events keep the beacon; visibilitychange takes the normal path, which creates the row first and which the page is still alive to await. tests/studio/sim_thread_settings_portability.py covers the migration against a populated pre-existing database, idempotency, the COALESCE preservation, unicode and emoji and Windows-shaped paths through the column, WAL, and the SQLite floor. It is stdlib only, so it runs on Windows and macOS as well as Linux. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Address the Codex review: pairing window, write ordering, and two over-broad guards Six items from the review, all reproduced against the code at head first. The composer is live while a chat's settings are being read, and until they land the store still holds the OUTGOING chat's values. snapshotQueuedChatRunSettings captures permissionMode at send time and sends it as permission_mode for the whole run, so a send in that window really does run under the previous chat's level, and a rejected read left those values up indefinitely. The store now drops to the installation defaults for the duration of the read, which is the only honest thing to show for a chat whose own settings are not known yet. Edits made during that window were still held, but the switch-away path replayed them into the installation defaults, which is the leak the holding exists to prevent. They are now committed to the thread they were made in; the values are still in the store at that point because the incoming chat's read has not resolved, and a chat with no row writes nothing, so an unsaved chat stays on the defaults. activeThreadScopedSettings was only refreshed when the debounce built its payload, so for 400ms after an edit neither it nor localStorage carried the new value. A model status poll landing there reverted the pill and the pending write persisted the revert. threadScopedOverride now prefers the live store while a captured write is pending; a pending pin keeps its own snapshot. Snapshot writes are chained per thread. The backend replaces settings_json rather than merging it, so two unordered writes do not merge, they pick a winner. Two guards were too broad. A chat first opened under Full access was pinned with no permission level at all and then followed the installation one forever; it now records the level underneath. And the deep research guard refused every external checkpoint, while externalCheckpointRefusesDeepResearch already exists and treats openai_codex as supported, which is the rule the composer follows. * Address the second Codex review: read/write ordering, the pre-hydration window, and two restore constraints - await this chat's in-flight snapshot PATCH before reading it back, so a chat edited, left and re-entered does not get its pre-edit snapshot applied over the values the user just set - keep the chat paired after a failed thread read, with a bounded retry, so later edits in it stay thread-scoped instead of moving the installation defaults - give each per-thread write a ticket so a queued write cannot land after the unload beacon and restore the older snapshot - start holding a chat's edits as soon as its id is known, not once /api/chat/settings has hydrated; the composer is interactive in between - keep Search and Thinking mutually exclusive on Kimi when restoring a snapshot - keep a stored Search/Code preference that a tool-less model has clamped off * Address the third Codex review: held-edit fidelity and the remaining write races - send only the edited fields, merged onto the row's own snapshot, when a chat's read never landed; the full replace was erasing settings the user never touched - release a held edit as soon as the first read says the row does not exist, instead of waiting for an unrelated history event - keep global hydration off a field whose edit is still held for its chat - keep a held edit out of the captured installation defaults - preserve a clamped Images and Fetch preference too, not just Search and Code - abort a settings PATCH that is already out when the unload beacon supersedes it * Address the fourth Codex review: settings merge semantics and a send that waits for them - PATCH gains settingsPatch, which applies only the fields it names. The unload path and the commit-on-leave path use it: both know what changed but not what else the row holds, and a replacement built from the defaults on screen was erasing the rest of the chat's snapshot - a replacement now keeps whatever the writing client could not read, so an older Studio opening a newer database no longer deletes the settings it had to drop - flush an edit held during pairing on a terminal event, keepalive, instead of relying on effect cleanup that unload does not guarantee - a legacy Dexie fallback means the backend read FAILED, so retry and keep the chat paired rather than releasing its edits to the installation defaults - park a send while the chat's own settings are still on their way, reusing the existing wait-and-send path, so a chat stored as ask cannot run on a global off * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Address the fifth Codex review: order snapshot writes on the server, and stop parking sends forever - give up openly when the thread read is out of retries: staying paired held every send behind a wait with nothing left to resolve it, so the chat now falls back to the installation defaults and says so - record the installation defaults when pairing begins. Deleting a held field from the capture left it with no fallback, and on the session's first pairing the edited value then stayed live into the next chat, which is the same leak - add settings_seq and refuse a snapshot write older than the one already stored: aborting a fetch does not stop a handler the server has already started - do the read, merge and write of a snapshot in one transaction, so two tabs cannot both build a replacement from the same stale row - re-send with keepalive on a terminal event anything an earlier visibilitychange flushed normally but has not landed - settle the source snapshot before forking, or the fork copies the modes the chat had before the pill the user just clicked * Pin the pairing window invariants that keep getting broken Eleven assertions over the gap between opening a saved chat and its snapshot arriving: hold rather than release, capture the defaults up front, merge rather than replace, order every write, end the pairing when the read gives up. Each one has been broken at least once during review, and each break either leaks one chat's settings into another or loses an edit the user watched happen. Source assertions rather than a driven store, matching the sibling store tests: a .tsx barrel sits in the store's import graph, so it cannot be loaded in a bare node test. Checked against the earlier revisions, 10 of the 11 fail on the first of them and 4 on the most recent. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Address the sixth Codex review: order writes per writer, not by wall clock - settings_seq is now compared only against the same writer's own earlier writes, identified by a per-tab id. Comparing one machine's clock with another's meant the browser that happened to be behind had every edit refused while still being told it had saved, which is worse than the race it was added to fix - carry the snapshot write in the same transaction as the guarded metadata write, or a rejected expectedTitle returns 409 with the settings already committed - remember a tab-close write that could not be confirmed and replay it next session: a chat whose row is still being created answers 404, and the creation that follows knows nothing about the edit - keep a chat's stored reasoningEnabled=false when the model forces thinking on - drop the thread-scoped state in compare mode. Two threads share one composer, so no single snapshot applies; leaving the module pointed at the last single chat let a model load read that chat's pills back into the pair Four more invariants pinned, all five failing on the previous revision. * Address the seventh Codex review: the queue path, retried pairings, and pending pins - gate the prompt queue on this chat's settings as well. handleSubmit reaches the queue branch before the send guard, so a prompt could be queued snapshotted from the installation defaults still on screen - sample the pre-window defaults once per chat rather than once per read attempt: a retry runs with the held edit already in the store, so re-sampling took that edit for the installation default and leaked it into the next chat - let a pending pin snapshot answer threadScopedOverride. A chat being pinned has no stored snapshot, so the override read null and a model load put the global pills back over the edit the queued write was about to persist Three more invariants pinned, all three failing on the previous revision. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Address the eighth Codex review: four ways a run or a write could still go wrong - sample the pre-window defaults from the installation defaults, not the store. On a switch from one saved chat to another the store still holds the outgoing chat's pills, so its values were becoming the default every snapshot-less chat follows - keep a write watermark per writer. One writer column meant another tab's write replaced it, and the delayed request the ordering exists to refuse was then compared against a watermark that was no longer its own - bound the thread read. It gates sending, and the underlying GET has no timeout, so one that never answered parked every send in the chat with nothing to release it - wait for the chat's settings in the adapter, which every run reaches. Reload, Continue and send-from-edit never touch the composer's guard and could start a run on the installation defaults - keep a replay entry until its PATCH answers ok; authFetch resolves for the 404 this exists to recover from - track a debounce-fired write as unsettled too, so a terminal event can resend it - settle a held edit before forking, not just the debounce and the write chain Seven more invariants pinned, all seven failing on the previous revision. * Address the ninth Codex review: bind the run's wait to its chat, order the replay - await the pairing for the RUN's own chat. One promise for all of them meant a run started for A was released by B's pairing ending, and then read B's settings for A - keep last session's replay ahead of this session's writes. It carries the previous writer id, so nothing on the server orders it, and on a slow link a full snapshot from last time could land after an edit made now and revert it - track unsettled writes by identity. Two ordinary edits both carry a null snapshot, so the first request's settle was deleting the second one's tracking - do not resend an older unsettled snapshot for a chat the terminal flush has just beaconed; every beacon takes a higher seq, so the stale one would win - evict writer watermarks by last use, never by counter: every session starts at 1, so comparing across writers threw out the newest tab and kept long-dead ones - stop clearing the defaults sample when committing held edits, which put back the resample-on-retry bug fixed a round earlier - let an explicit clear beat a capability preservation, so enabling Search does not quietly keep Deep Research stored as on - bound the gating read at the fetch, so a stalled GET is aborted and not just raced Nine more invariants pinned, all nine failing on the previous revision. * Make the per-chat settings Playwright test unload any resident model first The Search and Code pills are disabled while a model that cannot run tools is loaded, and this test drives both. The CI job it was added to leaves a small GGUF resident from an earlier step, so every pill click timed out on a disabled button: the test failed for a reason that has nothing to do with what it checks. Unload first, which is also the state the test is about, since with nothing loaded the pills stay pre-selectable. Found by running the job on a staging repo while the org queue was backed up; this step has never actually completed on this PR. * Address the tenth Codex review: release pairing gates one chat at a time - release a chat's pairing gate only for that chat. The previous round held them per chat but resolved every one on any settle, so a run started for A was still freed by B's pairing ending. Leaving a chat mid-read now leaves its gate shut, since its snapshot never arrived, and the wait is bounded so a run cannot hang on it - clear a thread's replay entry once a write of this session's lands. The entry carries the previous session's writer id, which the server will not order, so a retry could revert settings changed since - bound each replay request. Every settings write waits on them, so one socket that never settled left the session unable to persist anything - keep a failed write tracked, so a terminal event still beacons it. Dropping it left nothing to resend and the edit came back reverted - let a failed snapshot write reach the fork, which now stops rather than making a fork that carries the pre-edit modes Six more invariants pinned, all six failing on the previous revision. * Do not open a pairing window for a chat that cannot have a row An unsaved chat carries an assistant-ui runtime id (__LOCALID_...), which no row exists for, so its settings read can only 404. Holding edits behind that certain failure meant a pill or permission level chosen on a fresh /chat did not reach the installation defaults until the round trip came back, and playwright_chat_ui asserts it is there immediately: 'Run automatically persisted ask, expected off'. That test is untouched by this PR and passed at an earlier revision of it, so this is a regression the branch introduced. Found by running the suite on a staging repo while the org queue was backed up. * Keep a chat's own settings when its pairing or its write does not land Four fixes from review: - A run whose pairing wait ran out no longer proceeds. The wait only expires for a chat left mid-read, whose gate is held shut on purpose, so the store describes a different chat by then; the run is refused instead. The wait is also raised past the read's own retry budget so an ordinary slow read never reaches it. - A fork now fails when the flushed write for the chat it is copying failed. The replacement path reports failure by resolving false, which the settle helper ignored, so the copy took the pre-edit snapshot silently. - A replay clears only the body it sent. A terminal event in this session can store a newer body for the same thread while an older replay is out. - A provider constraint no longer rewrites what a chat stored. Kimi's builtin search cannot run with thinking, so the composer moves the other pill with persist: false, deliberately bypassing the capture path; the next full snapshot then saved the provider's value over the user's. * Pin the installation permission level the per-chat test compares against The UI workflow boots this server on the Studio home the chat-ui and cross-browser permission tests have already driven, so the install is shared rather than fresh and arrives carrying whatever level they left in the mirrored settings. Every assertion in the file names a literal level, so the run failed on staging with the composer showing Run automatically where Approve for me was expected, before a single per-chat assertion had been made. Set the default from the unsaved chat the test already starts on, where no chat is open and the edit is the installation's, and print it. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Bound the settings reads and recover the defaults hydration had to skip Four fixes from review: - Each pairing read now carries its own deadline and controller. bounded: true is the 30 second WRITE timeout, so the losing side of the 8 second race stayed open while the retry opened the next one, up to three per chat during an outage, and leaving the chat cancelled none of them. - The ensure step in front of a settings write is bounded and takes the caller's signal. It runs before the write, so neither the signal nor the write timeout reached it, and a stall there left the per-thread chain pending for the life of the page with reopening and forking that chat waiting on it. - A default hydration skipped because its field was held is kept and used when the pairing window closes. The restore was falling back to the pre-hydration copy, so snapshot-less chats followed this browser's stale value rather than the server's until a reload. - Tab-close snapshots are replayed even when /api/chat/settings fails to hydrate. They are rows' own settings and have nothing to do with that endpoint; the replay is guarded so it still runs once. Also documents why a refused settings write does not roll the metadata in the same PATCH back: a refusal means this writer already landed something newer, not that the write failed. --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Daniel Han <danielhanchen@gmail.com>	2026-08-16 00:13:04 -07:00
Michael Han	eb63760b6f	Studio: find the real Documents folder on Windows for project workspaces (#8955) ... * Studio: find the real Documents folder on Windows for project workspaces Project workspaces are the only thing Studio writes to Documents, so a Documents folder resolved wrong breaks project creation and nothing else. OneDrive's Known Folder Move repoints it and leaves ~/Documents behind. Also report which folder could not be created, instead of a bare 500. Fixes #8936 * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: only blame the projects folder for a projects folder failure The same upsert opens studio.db first, which is a different path. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: stub the workspace refusal instead of staging it with chmod Root ignores a read-only directory and Windows does not enforce one. --------- Co-authored-by: Unsloth <michaelhan@Michaels-MacBook-Pro.local> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-08-15 22:48:24 -07:00
Maheswar Kumar	c38daa3b5d	studio: carry chat settings across browsers and remote sessions (#8656) ... * carry chat settings across browsers and remote sessions 26 chat settings were written only to localStorage, so opening Studio in another browser, a private window, or a remote session started every one of them from defaults: the web search, code, image and web fetch pills, MCP, deep research and its website policy, artifacts and the canvas menu item, the permission mode and confirm-tool-calls toggle, all eight RAG knobs, the speculative decoding preference, the GPU memory mode, and the model list quantization preferences. These describe the installation, not the browser holding them, so they now round-trip through /api/chat/settings alongside the inference params and presets that were already stored there. saveBool, saveString, saveRagSource, saveResearchWebsitePolicy and savePermissionMode route through one persistSetting, which mirrors a changed value to the backend. localStorage stays the synchronous boot cache: hydration writes the server values back into it, and any setting the server has never stored is backfilled from it, so an existing install seeds itself on first load instead of waiting for each toggle to be touched again. Mirror writes are held until hydration has run, otherwise the model-load path would push a fresh browser's defaults over what another browser already saved. Full access stays session-only. The backend rejects permissionMode "full", savePermissionMode never writes it, and hydration leaves both permissionMode and confirmToolCalls alone while it is active, so a stored level cannot drop a sandbox bypass the user accepted a warning for. ChatSettingsPayload is extra="forbid" and a single out-of-contract field 400s the whole save, so mirrored-chat-settings.ts re-validates every value against the same literals and ranges before it is sent. Its bounds match the backend: ragTopK 1..50 as /api/rag enforces, ragAutoInjectMinScore 0..1, domain names capped at 253 characters. tsconfig.test.json now includes src/speech-recognition.d.ts: the new test reaches a src module that relies on those ambient globals. * close four gaps in the mirrored chat settings Hydrate speculativeType and gpuMemoryMode into the store, not just the storage cache. Both are store fields the settings sheet and use-active-model-config read, so writing only localStorage left a fresh browser showing and applying auto while the installation had off or manual, until something reread storage. They join the scalar settings, which removes the storage-only path entirely. Replay a mirrored edit made before the initial GET lands. The version bump now happens on the write and the patch is held, then flushed after hydration, so a toggle flipped during startup is no longer reverted by the arriving server value and then skipped by the backfill. Backfill the derived legacy values. A profile predating permission levels holds only unsloth_chat_confirm_tool_calls, and one predating the three-way RAG auto-inject control holds "true"/"false"; neither carried, because the backfill required a raw value under the current key and sent the legacy string as is. permissionMode now seeds from loadPermissionMode, and the auto-inject codec runs the same migration loadRagAutoInject does. Replace ragSource instead of merging it, in both the client patch queue and the backend deep merge. It is a discriminated union, so switching from a knowledge base back to the thread source produced {type: "thread", kbId: ...}, which the payload's thread variant forbids and which the merge would otherwise leave in the database. * separate load-applied settings from user edits, and hydrate at the root A model load that finishes before the initial settings GET calls saveSpeculativeType and persistGpuMemoryModeOnLoad with whatever the cache returned. Queuing those as pre-hydration edits bumped their mutation versions, which blocked a stored ngram or manual value from hydrating, and then replayed the load's default over the backend. Both now go through persistLoadDerivedSetting, which writes only the cache until hydration has run. Every remaining caller of saveString and saveBool is a user action. hydratePersistedSettings now runs from a root-level mount. Opening Studio straight at /hub never mounted ChatPage, so the Models page read and wrote fitOnDeviceOnly, expandQuantizations and showAllQuantizations against a store that never hydrated: a stored value was not applied, and a toggle only reached the pre-hydration queue, which the beforeunload flush does not cover. showCanvasMenuItem joins permissionMode in backfilling a derived legacy value. A profile predating the visibility flag keeps Canvas shown through an explicit plus-menu pin, and loadShowCanvasMenuItem reads it, so requiring a raw value under the new key hid Canvas on every other browser. * send held startup edits when the tab closes The beforeunload flush covered pendingPatch only, so an edit made while the initial settings GET was still in flight died with the tab: the next session hydrated the older server value over the local cache and skipped the backfill, because the field already existed on the server. drainPreHydrationPatch folds the held edits onto the outgoing patch first, so the keepalive PUT carries them. Load-applied writes are unaffected, since persistLoadDerivedSetting never queues them. * rerun the external-model pill normalization after settings hydrate The effect that clamps the tool pills to the selected external model writes the store directly, so it bumps no mutation version. When it ran before a slow settings GET, hydration then restored the persisted pills over it: Kimi came up with search on despite the deliberate off default, and a model without a given builtin could be re-enabled outright. It now depends on settingsHydrated. Hydration applies the server values and refreshes the localStorage copies the effect reads, then the effect reruns and clamps them to the model, so the model-aware result is the one that lands. * keep deep research clamped for external models, and let an atomic key repair a corrupt row setCheckpoint sets deepResearchEnabled false for an external model id (chat-runtime-store.ts 2301). Restoring the last external checkpoint runs that clamp during startup, and hydration then landed after it and re-armed the flag from the server, so the composer showed the pill active and the send failed on chat-adapter's requirement of a local model or an openai_codex provider. Before this branch the localStorage value was read at store construction, which is before setCheckpoint, so the clamp won; hydration is what reordered it. Hydration now skips a stored true while the active checkpoint is external. upsert_chat_settings_merge refuses a dict patch aimed at a quarantined key, because the base it would merge onto is gone. ragSource is an atomic whole-value replacement, so that guard turned a valid new selection into a 409 and left the pick unsaved. Atomic keys are exempt; partial patches still conflict. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * clamp deep research only for external providers that cannot run it The previous clamp keyed on isExternalModelId, which also covers openai_codex. Codex runs deep research: thread.tsx derives researchDisabled from every external provider except that one, and chat-adapter accepts its research requests. Restoring a Codex checkpoint therefore left the store at false while the cache said true, losing the mirrored preference. externalCheckpointRefusesDeepResearch resolves the provider behind the checkpoint and clamps only a known non-Codex one. An unresolved provider is left alone, since the connection list may not have loaded yet and refusing on that would drop the preference for the same reason. * close three more pre-hydration ordering gaps A model load captures its settings up front and persists them on success, so one that started before hydration and finished after it wrote the stale capture over the preference that had just arrived. saveSpeculativeType and saveGpuMemoryMode now pass whether the value still matches the live store, which every path that sets loadSpeculativeType or loadGpuMemoryMode keeps in step, and a stale write is dropped rather than mirrored. An explicit write that matched the local cache was recorded nowhere, because persistSetting compared against storage before deciding to mirror. Before hydration the cache says nothing about the server, so enabling deep research turned the tool pills off locally, sent only deepResearchEnabled, and hydration then restored toolsEnabled: both mutually exclusive modes ended up on. Writes made before hydration are recorded whether or not they change storage. resolveVisionOverrides awaits hydration. It reads ragOcrScanned and ragCaptionFigures for every RAG upload path, and on a fresh browser it returned undefined before the settings arrived, so the backend ran its default OCR and caption passes on an ingest that cannot be undone afterwards. * keep an always-on model's reasoning on through hydration A local load sets reasoningEnabled from the model's own capability (use-chat-model-runtime.ts 1404, apply-inference-status-to-store.ts 343), by writing the store directly, so it bumps no mutation version. A load that finished before a slow settings GET then had a stored false applied over it, and the adapter sent a real disabled-thinking request to a model that cannot honour it. The external normalization effect does not cover this: it returns for a non-external checkpoint. Hydration now leaves reasoningEnabled alone while reasoningAlwaysOn is set. Only a load sets that flag, so the guard cannot fire before a model is known, which is why it keys on the always-on case rather than on supportsReasoning, whose default of false says nothing until something has loaded. * bound the settings hydration an ingest waits on resolveVisionOverrides now awaits hydratePersistedSettings before every upload, and that await has no deadline of its own: authFetch sets no AbortSignal, so a wedged or unreachable server leaves the promise pending as long as the socket does. Uploads read these overrides synchronously before this branch, so the drop would hang with nothing on screen rather than fall back. Race the hydration against an 8s wait, matching awaitFirstSave in chat-adapter. On the timeout the local values are used, which is what a blocked localStorage already falls through to. * fix the permission-mode UI test for a server-backed level The migration cases mutate localStorage and reload, which only decides the pill while the install has never stored a level. Once one is stored, hydration applies it and rewrites the local copy, so the second reload asserted against the level the first one seeded and the step failed on Linux, Windows and macOS. Refuse the hydrating GET for the length of the loop, which is the state the derivation actually governs, and add the other half of the contract: with a level stored, a browser holding only the legacy key gets the installation's level back. * hold the permission block's fresh-profile state across browsers The cross-browser step runs this block three times against ONE install (.github/scripts/run-studio-permission-browser.sh wipes auth, not the settings), so with the level stored per install runs two and three opened on the level run one left behind and the fresh-profile assertion failed on Linux, Windows and macOS alike. Extend the refused hydration back over the opening assertions and clear the local level first, which together are the state a first-ever browser on a never-configured install is in. * Keep hydration from desyncing the loaded model shadows for PR #8656 * Stop one rejected settings patch from blocking every later save The settings queue requeued a failed patch as it stood. /api/chat/settings is extra="forbid" and refuses the whole body on one bad field, so a patch the server will never accept came back on every following save and took it down too: after a single 400 the tab could no longer persist any chat setting, and nothing retries until the next edit or the tab closes. Reachable without a bug on either side. A new bundle against a rolled-back backend has every mirrored field rejected as extra_forbidden, which is exactly that shape. Measured against a live Studio in Chromium 151, Firefox 153 and WebKit 26.5: after one rejection every later request still carried the rejected field and no unrelated edit ever went out clean, in all three. A failure is now classified before the patch is requeued. 5xx, a network error, 408 and 429 keep the patch. Any other 4xx is the server saying this body is wrong, so the fields it named in the validation detail are dropped and the rest of the coalesced patch is kept and rescheduled, which cannot loop because each round strictly shrinks the patch. An unattributable 4xx drops the patch outright rather than guessing. Two smaller ones alongside it, both measured: A non-finite number reached the database. json.loads accepts bare NaN and Infinity and pydantic took them for a float, so {"temperature": NaN} was written into value_json as a bare NaN token. Python reads that back, so the row was never quarantined, while the response model rendered it as null: the value was silently lost and the row on disk was not valid JSON for any reader that is not Python. The settings models refuse non-finite numbers now, the way models/training.py already does for every numeric training field. The 400 for one of those was a 500. detail = exc.errors() echoes the offending input, and Starlette's JSONResponse dumps with allow_nan = False, so a rejected NaN made the error handler itself unrenderable and the caller got an unhandled ValueError for a request the validator had correctly refused. This route now uses safe_validation_errors, which the repo already has for exactly this and which also bounds a multi-megabyte value being quoted back. The unload flush also runs on pagehide and on the hidden transition of visibilitychange. beforeunload is not the end of a page's life on mobile Safari or a discarded Android tab, and a page restored from the back/forward cache never unloaded at all. Safe to add rather than replace: the pending patch is swapped out before the request, so a second call with nothing queued returns immediately. keepalive is now skipped above 60 KiB, since Fetch caps in-flight keepalive bodies at 64 KiB and a valid researchWebsitePolicy (2000 domains of 253 characters) is well past it; over the budget the request fails immediately in all three engines, where without keepalive it at least has a chance, and on the visibilitychange flush it simply succeeds. Tests: 11 new frontend tests over the retry policy and the keepalive budget (2042 pass), 6 new backend tests over non-finite numbers and the renderable 400 (114 chat backend tests pass), tsc and ruff clean. * Let the loaded shadow decide what hydration owns for PR #8656 * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: danielhanchen <danielhanchen@gmail.com>	2026-08-13 17:33:28 -07:00
omjha3125-ctrl	5e184f50c6	Studio: allow max output overrides for custom providers (#8512) ... * Studio: allow max output overrides for custom providers * Studio: fix custom Max Tokens override breaking unrelated connection edits Three problems found while driving the override end to end against isolated Studio builds. 1. Every save of a connection Studio DISPLAYS as Custom but STORES as `openai` returned 400. `resolveUiProviderTypeFromConfig` maps a row saved as `openai` with a custom display name or base URL back to UI type `custom`, so the new Max Tokens limit row rendered for it, and `parseMaxOutputTokens("")` returns null rather than undefined, so the key was still sent and the backend, which keys the contract on the STORED type, rejected it. A plain rename, a model change or a key rotation all failed with an error about a field the user never touched. The backend now accepts an explicit null on any provider type (clearing an override that cannot exist is a no-op) and the dialog decides from the stored type, carried on `ExternalProviderConfig.backendProviderType`. 2. The new clamping effect could silently lower and PERSIST a configured Max Tokens. `isExternalModel` comes from the checkpoint string while the cap comes from the resolved provider, and the two disagree whenever the provider is momentarily unavailable: connections toggled off, a cold browser where settings hydrate before the provider sync lands, or a connection disabled while selected. The cap then collapsed to the 32,768 fallback and the effect wrote it through, permanently, which defeats the feature it belongs to. It now waits for `settingsHydrated` and a resolved provider: no provider means the cap is unknown, not 32,768. 3. The Max Tokens limit field is a text input rather than `type="number"`. The HTML value sanitization algorithm replaces anything the engine does not read as a valid floating-point number with the empty string, and empty means "no override" here, so a grouped entry like "131,072" left the box looking filled and silently cleared the saved override. Observed in Firefox, where a stray character reported "" and the save reported success; Chromium and WebKit kept the digits. The raw string now reaches the existing validator, which rejects it with a reason. This matches NumericValueInput in the run-settings panel. The two decisions are pure functions, `supportsCustomMaxOutputTokens` and `resolveExternalMaxTokensClamp`, so they can be tested without a DOM. * Studio: cover the custom Max Tokens override's upgrade and contract paths Backend: an existing studio.db written before this column exists, opened by this build and then opened AGAIN by a build that has never heard of the column, which is what reverting to the previous release does; migration idempotence, since ALTER TABLE has no IF NOT EXISTS; the route contract per provider type, where an explicit null has to be accepted everywhere and a real override still refused outside generic Custom; the set, preserve and clear lifecycle asserted against the stored row; exact round-tripping at the top of the accepted range; and the metadata rollback when a credential write fails. `test_credential_routes.py`: the test that pinned "reject the field on any non-custom provider" now pins "reject only a non-null value", plus two tests asserting an explicit null is accepted on create and on update. Frontend: the stored-versus-displayed provider type truth table including the unknown-type fallback, normalization of anything an older localStorage entry could hold, a golden slice of the capability table proving no named provider's cap moved, the clamp guards, and a pre-feature entry loading without gaining a cap. Verified against three isolated installs (merge base, this branch before these fixes, and after) on Linux with chromium, firefox and webkit, which covers Chrome, Edge, Firefox and Safari's engine. An old home upgraded in place keeps its connections, migrates its schema and reads the 32,768 fallback; the same home opened again by the previous build still lists, edits and creates. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: guard the remaining Max Tokens clamp sites on a resolved provider Two more places applied a cap derived from provider data that may not be there yet, and both only ever lower, so a blink in the provider list cost the user their configured value permanently. `applyPresetParamsWithinCurrentLimits` was gated on `isExternalModel` alone, so applying or falling back to a preset while the connection was unresolved wrote the 32,768 fallback into the live params. `setCheckpoint` looked the provider up and clamped whether or not it found one, so a checkpoint restored or selected before the provider store hydrated read the fallback instead of the connection's cap. That path predates this feature, but the override is what gives it something to destroy: without one, a Custom connection could never exceed 32,768 and the clamp was a no-op. Both now take the rule the settings-panel effect already uses: no provider means the cap is unknown, not 32,768. * Studio: tighten the comments added with the Max Tokens override * Studio: keep providers_db importable on the declared Python floor `max_output_tokens: int | None | object = _UNSET` is the module's first PEP 604 annotation, and annotations evaluate at def time, so the module stopped importing on 3.9 while pyproject still declares >=3.9. tests/test_python39_compatibility.py enforces this with a debt counter and caught it at 36 files, up from 35. --------- Co-authored-by: danielhanchen <danielhanchen@gmail.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-08-13 04:22:31 -07:00
Wasim Yousef Said	ef8daca43e	Unsloth Studio: add ChatGPT subscription chat with Codex tools (#8511) ... * feat(studio): add ChatGPT Codex subscription backend * feat(studio): add Codex subscription chat experience * feat(studio): support Codex deep research runs * fix(studio): harden Codex chat integration * fix(studio): complete Codex subscription integration * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix(studio): release Codex credential locks across threads * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix(studio): complete the Codex browser callback * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix(studio): address Codex tool and auth review * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix(studio): harden Codex tool and research flows * fix(studio): close remaining Codex review gaps * fix(studio): address major Codex CI findings * fix(studio): enforce Codex tool and provider policy * fix(studio): show OpenAI icon for ChatGPT connections --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-08-12 14:26:33 +02:00
Wasim Yousef Said	19abc08bb4	Studio: persist provider and Hugging Face credentials (#8299) ... * Studio: persist provider and Hugging Face credentials * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: address credential persistence review * Studio: preserve account-bound legacy credentials * Studio: make persisted credentials installation-wide * Studio: address credential review follow-ups * Studio: address final credential review * Studio: close credential lifecycle races * Studio: make credential migration race-safe * Studio: reconcile delayed HF credential writes --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-08-11 11:06:20 +02:00
Daniel Han	3c77be4463	Studio: keep RAG lexical search off the linked-folder filters when nothing is linked (#8394) ... * Studio: keep RAG lexical search off the linked-folder filters when nothing is linked The visibility predicates added with linked folders join chunks and documents and evaluate two correlated subqueries for every row the FTS match returns, before the LIMIT, so their cost grows with how common the query terms are rather than with k. On an install with no linked folders and no retired scopes they cannot exclude anything, so run the plain FTS query there instead, gated on two EXISTS reads of the two tiny tables. Also reuse the search handler's own RAG connection for its ownership check rather than opening a second one, since sqlite-vec loads per connection. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Read the fast-path gate and the FTS match on one snapshot The gate and the search ran as two autocommit statements, so a scope retired between them was invisible to the gate and visible to the read: the plain query then returned rows the retirement predicate would have dropped, and hydration discarded them after they had taken slots under the LIMIT. A deferred transaction pins both reads to the snapshot the gate saw. A caller that already holds a transaction keeps its own. * Tighten the comments this branch adds * Gate the lexical fast path on what the filters can actually hide Two ways the gate read the wrong thing. A purged tombstone is kept for the life of the database, so deleting any knowledge base, linked folders or not, left the gate true for good and the fast path never ran again; only unpurged tombstones can hide a row, and a purged scope has no documents left. And a folder-owned document whose mapping row never landed outlives its folder row on unlink, so the gate now counts those documents directly, through a partial index that is empty when nothing is linked. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Tighten the comments this branch adds --------- Co-authored-by: danielhanchen <elliegouldingstuff@gmail.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-08-11 00:36:24 -07:00
oobabooga	0e488c211b	Studio: fix delayed chat streaming for already-loaded GGUFs (#8371) ... * Studio: keep local model scans off chat requests * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: harden scan index invalidation * Studio: preserve invalidated background scans * Studio: publish scan trust atomically * Studio: skip scans after no-op folder removal * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Bound additions-only cache trust and keep the warm slot releasable for PR #8371 - Encode the additions-only invalidation as a negative monotonic stamp instead of a constant sentinel, so retained entries stop being trusted once the background rebuild can no longer be running. A rebuild that keeps failing previously left them trusted for the life of the process, and a model deleted on disk out of band could still trigger a switch that then fails at load. - Release the warm slot in a finally. A BaseException out of the scan left _warming set forever, which killed background warming and put the scan back on the request path, the exact latency this PR removes. - Skip the freshness shortcut in _index() and warm_index_soon() when the stamp is not positive. monotonic() counts from boot, so on a host whose uptime is under the TTL an invalidated stamp read as recent and served what was just revoked. - Note in _loaded_identity_satisfies that a request naming the load path itself skips the alias recording, which the previous wording implied it did not. * Age the resolver clock instead of the stamp in the trust tests Both tests built an aged snapshot as -(time.monotonic() - age). monotonic() counts from boot, so on a host whose uptime is below that age the subtraction goes negative and the negation yields a positive stamp, which reads as an ordinary fresh scan rather than an aged additions-only one. GitHub's macOS and Windows runners are fresh enough to hit this, so both failed there while passing locally. Advance the module's monotonic clock instead, which is what a real process does and carries no assumption about uptime. Also patch the module's own time reference rather than the real time module, so a stray warm thread sharing it does not get a frozen clock. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Record the advertised alias when a request names the load path for PR #8371 The resident short circuit matches on model_identifier, which is the on-disk path for a manual local load. A request naming that path answered from it and skipped _record_serving_alias, so a loose .gguf whose scanner alias is not its filename was advertised, and reported in every response, as the filename. Hold those requests back until the alias has been recorded. Only the first one pays the resolver: every later request matches the recorded alias and takes the short circuit, so the streaming win is unchanged. * Pin the resident shortcut to the resident check it must not outrun The shortcut added here is the only path that answers a request without consulting the filesystem index, so if it ever accepts a name the pre-existing resident check would reject, the wrong weights get served. Assert the implication directly over a matrix of resident identities, requested names and quants: anything the shortcut accepts, _loaded_satisfies accepts too, which makes it a strict subset of the previous behaviour and leaves a miss costing only the scan it used to cost anyway. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Say the warm-slot handoff once The module comment on _warm_pending already states the contract, so the block at the call site only needs to say why this code path can be the one that trips it. --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: danielhanchen <danielhanchen@gmail.com> Co-authored-by: danielhanchen <danielhanchen@users.noreply.github.com>	2026-08-10 23:49:30 -07:00
alkinun	86b3ee1853	Studio: sync linked folders into RAG (#8014) ... * Studio: sync linked folders into RAG * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix linked folder lifecycle edge cases * Fix linked folder sync review issues * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix linked folder sync lifecycle * Fix linked folder sync review findings * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Handle linked folder availability and KB cleanup * Harden linked folder filesystem reconciliation * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Retire project RAG scope before deletion * Skip project RAG cleanup when unavailable * Stabilize linked folder reconciliation * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Guard linked folder deletion races * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Guard project RAG uploads during deletion * Fix linked folder scope deletion recovery * Fix deletion transaction and native lease calls * Avoid Studio write locks during folder sync * Harden linked folder startup and scans * Preserve linked folder scans with weak identities * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Make linked folder reconciliation crash-safe * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Preserve linked folder lifecycle intent * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Make linked folder retirement cleanup durable * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Close linked folder lifecycle races * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Close RAG visibility and lease gaps * Coordinate durable RAG workers * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Refine linked folder lifecycle and reconciliation Preserve signed folder identities through registration and revalidate them before persistence. Use durable scope tombstones, owner-first project deletion, file-first cleanup, and lease-based job recovery. Run linked-folder ingestion within the folder worker and retain prior mappings when reconciliation is incomplete. Remove compatibility paths for unreleased linked-folder schemas and reuse the shared scoped single-flight request helper. Consolidate regression coverage while preserving lifecycle, concurrency, and cleanup scenarios. * Align Windows path identities and preserve leased RAG jobs * Encode 128-bit file identities for linked-folder mappings in PR #8014 CPython 3.12+ fills st_ino from FILE_ID_INFO, so on ReFS and Dev Drive volumes a file id is 128-bit and does not fit SQLite's signed 64-bit INTEGER. The root identity was already hex-encoded through _store_identity; the per-file mappings still wrote st_dev/st_ino raw, so _install_mapping raised OverflowError for every file and nothing on those volumes could ever be indexed. Route the per-file identity through the same encoding and compare change detection on the encoded pair, so existing integer rows keep matching. * Reap linked-folder document orphans during periodic scheduling _recover_startup_state() runs once, before the worker loop. When a second backend crashes between a completed ingestion and _install_mapping(), the surviving process reclaims the expired folder job and reindexes the file, but its own startup pass is long past, so the earlier document, its chunks and its snapshot stay unreferenced forever and keep answering searches alongside the replacement. Extract the orphan reap and run it from _enqueue_periodic() as well. The live ingestion and folder-sync lease guards are unchanged, so work another backend still owns is left alone until its lease expires. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Remove replaced linked-folder snapshots only after the mapping commits _install_mapping() unlinked the previous snapshot inside the transaction. A commit failure, for example SQLITE_BUSY under the multi-process contention this feature already handles, rolls the mapping and the document deletion back but cannot restore the file, leaving the prior document searchable while preview and download fail on a stored_path that no longer exists. The delete paths keep file-first ordering on purpose, since there the surviving row is the retry queue. Here the surviving row is live data, so the removal now runs after the commit through _remove_snapshot(), which logs rather than failing an installed mapping. * Release skipped folder-sync claims and keep the worker alive on queue errors _next_job() claims and activates a job before dispatching it, and reconcile_folder() then re-claims. When an unlink lands in between it fails the job, the second claim returns None, and the early return skipped the finally that releases the lease. The heartbeat renewed that claim every 5 seconds for the process lifetime, so _delete_retired_folder() never saw it expire and delete_folder() spun on its 50 ms wait until restart. Separately, _next_job() ran outside any retry block, so a writer-lock timeout unwound the worker through the outer finally and left nothing to relaunch it, stopping every linked-folder scan for the process lifetime. It now backs off and retries like the initialization and periodic paths already do. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Defer deleted-file snapshot removal and release exhausted folder polling _delete_mapping() unlinked the snapshot inside its transaction. A failed commit rolls the mapping, document and chunks back but cannot restore the file, so the document stays searchable while preview and download fail. Unlike the retired folder queue, which _reconcile_retired_folder_deletions() retries every cycle, nothing re-drives this path on a folder with auto_sync off, so the broken state can persist until someone syncs by hand. Removal now runs after the commit, as _install_mapping() already does. The folder job poller also only released its controller from the catch block, so exhausting all 600 attempts left the id in controllers.current; the refresh then treated the job as tracked and never restarted polling, freezing progress until unmount. * fix linked-folder deletion, sync coalescing and buffered progress streams Six defects found while exercising the linked-folder pipeline end to end. Deletions no longer depend on a clean ingest. `_reconcile_folder` gated its delete pass on `failed == 0`, so a single permanently unparseable file kept every removal from ever applying: a document the user deleted from the folder stayed indexed and kept coming back in retrieval, with no way out, since managed documents cannot be deleted by hand. Each vanished path now gets one grace pass, recorded in `linked_folders.withheld_paths`, which preserves the guarantee that a rename or replace keeps its prior document while its replacement fails to index. A sync requested during a running sync is no longer dropped. `_request_sync` folded it into the running job and returned that job's id, but that job had already fixed its file list, so the change was never indexed and the user was shown a completed sync. `rebuild_requested` becomes `successor_kind`, one column covering both kinds, and `_fold_into_active_job` queues a follow-up whenever the running job cannot cover the request. Folder-sync progress survives a reverse proxy. A Cloudflare tunnel buffers the whole event stream until the response ends, which no origin header, padding or keepalive prevents, so the UI sat at zero for the entire sync. `readSseJsonEvents` abandons a stream that goes silent for 12s and the caller reconciles by polling, and `job_events` now emits a keepalive so only a genuinely buffered stream trips it. Also fixed: the job error names the files that could not be indexed instead of only counting them; `Content-Disposition` carries the base name rather than the folder-relative path; and `rag_home` places its studio home outside the macOS denylist, where `/private/var/folders` made 48 of the 67 linked-folder tests fail. Refactoring: `streamJobEvents` and `streamFolderSyncJobEvents` share one reader, `sse-framing.ts` moves to `src/lib` so its frame splitter is the only one, and `ensure_linked_folder_columns` upgrades both the vector and metadata connections. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * do not retire a project rag scope that was recreated during delete The row delete commits before the workspace work, so another client can create a project with the same id in that window. Retirement writes a permanent tombstone, which left the new project unable to link folders or upload documents at all. Retirement now rechecks ownership under the scope lock, and periodic reconciliation drops the tombstone of any scope whose owner exists again, so a scope retired in the remaining race recovers instead of staying dead. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * check project ownership under the scope lock before periodic retirement The listing and the ownership check ran outside the lock that create_folder and upload admission take, so a project recreated with the same id could link a folder that periodic reconciliation then marked retired with auto_sync off. Dropping the tombstone afterwards never restored those fields, leaving the folder unusable. The check now happens inside the scope lock, so a link either precedes retirement and blocks it, or is rejected by the tombstone. * bound scope retirement to folders linked before the ownership check The project rows live in studio.db and the folder rows in rag.db, so the ownership check and the retirement write cannot share a transaction. A second backend process sharing the same home can commit a folder link in that window, and retiring it left auto_sync off with no path back: dropping the tombstone never restored those fields. Retirement now only reaches folders created at or before the moment the scope was found ownerless, so a link from another process keeps its own state. Also close the source descriptor when the uploads root cannot be prepared. os.open succeeded before ensure_dir raised, outside the try that closes it, so a read-only or full uploads directory leaked one descriptor per file in the pass. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: oobabooga <112222186+oobabooga@users.noreply.github.com> Co-authored-by: danielhanchen <elliegouldingstuff@gmail.com> Co-authored-by: Maheswar Kumar <110882203+mahiatlinux@users.noreply.github.com>	2026-08-10 05:21:09 -07:00
Maheswar Kumar	5c332c323f	studio: render a chat send without waiting on persistence (#8136) ... * studio: render a chat send without waiting on persistence Sending the first message in a new chat cleared the composer, then left the thread blank for a second or more before anything appeared. assistant-ui's LocalThreadRuntimeCore.append awaits the thread-list item's initialize promise before adding the message to its repository, and the composer has already emptied by then. The Studio adapter's initialize wrote the thread row first, so the whole stall sat between the composer clearing and the first paint. createStudioDbAdapter.initialize now returns the thread id straight away and registers the row write through trackStoredChatThreadRecord. The user message and the assistant placeholder render in microtasks. The model run still waits for persistence through the existing history-append gate, so a failed write cannot leave an orphan run. ensureStoredChatThread awaits any in-flight row write, which keeps the guarantee at one choke point instead of asking every initialize caller to remember it. Without that, the prompt queue's model correction in thread.tsx silently dropped its patch: updateStoredChatThread returned undefined for a row that had not landed yet, and the thread persisted with the live global model rather than the model captured for the queued run. Three round trips also left the send path: - ensureThreadRecord answers "does this thread exist" with getStoredChatThread instead of listing every thread, so it no longer scales with how many chats exist - the history append reads the thread and the message together, and looks the message up by id rather than fetching the whole thread history - listStoredChatThreads re-reads after the legacy import only when that import actually created threads, tracked by legacyChatImportGeneration Handlers in routes/chat_history.py are sync defs so Starlette runs studio.db's blocking sqlite in its threadpool. As async defs they ran it on the event loop, where a thread listing serialised behind model loads and status probes. test_route_handlers_are_sync guards the conversion, and the tests that drove those handlers through asyncio.run now call them directly. Thread keyed GeneratedImageOverlayProvider on the app's activeThreadId, which flips from "default" to the thread id the moment the first message persists. That remounted the whole message tree and replayed the user bubble's animate-in classes just as the response started. It now keys on assistant-ui's threadListItem.id, which is stable across that transition and still changes on a real thread switch. Verified with ruff, 237 backend tests, 900 frontend tests, tsc, and biome. GPU checks were not run: none of this touches a model path. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * studio: close three races around the tracked chat thread row write Deleting a thread now waits for a row write that initialize() started, so an in-flight upsert cannot land after the DELETE and resurrect an empty thread that nothing later cleans up. Overlapping initializations chain in trackStoredChatThreadRecord instead of replacing the tracked promise. Waiting on only the newest upsert let a slower one finish afterwards and overwrite a modelId a queued run had already corrected. backfillLegacyThreadFields bumps the listing generation when it restores a container binding. Only the row-creation branch bumped it, so listStoredChatThreads skipped its re-read and returned the pre-backfill record, and the OpenAI container cleanup in openai-code-exec-section could miss a thread still bound to a deleted container. * studio: drain later thread row creators and bump the listing generation on commit awaitStoredChatThreadRecord read the tracked entry once, so a creator registered while it was already waiting was missed and a delete could still beat that upsert. It drains until the entry stops changing, reporting the last outcome so a thread nobody wrote is still the failure case. backfillLegacyThreadFields bumps the generation after the patch commits rather than before. A listing that captured the bumped value while the request was in flight read the old row, then skipped its re-read because the generation never moved again. * studio: gate thread document materialization on the row write ThreadDocumentsBar treated a resolved initialize() as proof the thread was persisted. Since initialize() now returns before the row write, a failed thread POST no longer reached the catch: the bar reported success, indexed documents against a thread id that was never created, and left the user without the error toast. Awaiting the tracked creation restores the previous behaviour for this consumer while the message path keeps rendering immediately. * studio: retry a failed chat thread row write on the next write A transient failure in the row write left the chat permanently broken. initialize() had already resolved, so assistant-ui cached the remote id and never asked the adapter for the row again, and ensureStoredChatThread can only read or import an existing row. Every later message write then returned 404 until the user abandoned the chat. trackStoredChatThreadRecord takes a creator rather than a promise and keeps it when the write rejects. ensureStoredChatThread re-runs it once the reads confirm the row is genuinely absent, which restores the retry that rejecting from initialize() used to provide. * studio: confirm the chat row before materializing thread documents Waiting on the tracked write was not enough. After a failed row write the tracked promise is already gone and assistant-ui returns its cached initialize() result, so a retried attachment resolved immediately and indexed documents under a thread id with no chat row. The bar now goes through ensureStoredChatThread, which reads the row and re-runs a failed creator before reporting success, so the failure reaches the existing catch and its toast. Temporary chats are exempt: they never persist a row, so isThreadIncognito skips the check. * studio: drain pending thread row writes before clearing history Only per-id deletion waited for a tracked row write, so clear-all could race one. The DELETE finished first and the POST recreated the thread, and because the inventory clearStoredChats builds comes from a listing taken before that POST landed, the thread was never tombstoned either and survived the clear outright. The drain runs before that listing so a freshly written row is both visible and removed, and it discards failed creators so a later retry cannot recreate what the user just cleared. * studio: retry a failed chat thread row write from ensureStoredChatThread ensureStoredChatThread awaits awaitStoredChatThreadRecord before its reads, and that rethrows a rejected row write. The retryFailedThreadRecord branch further down never runs for a caller that was already waiting when the write failed, so a recoverable write surfaced to it as a rejection instead. updateStoredChatThread is the visible one: a rename or an archive over a row write that had just failed rejected rather than retrying the creator and applying. The next call did recover, because the failed creator stays registered, but only after the first one had already thrown. Waiting on the tracked write is still correct, adopting its failure is not: the reads below already decide whether the row exists, and retryFailedThreadRecord re-runs the creator when it does not. * studio: bound the wait on a thread row write in delete and clear Deleting a thread and clearing history both wait for a tracked row write so an in-flight upsert cannot land afterwards and resurrect what was just removed. Neither wait had a bound, and authFetch sets no timeout or AbortSignal, so a request the backend never answers blocked both for as long as the browser took to give up. Before the drain existed there was nothing to wait on, so this is reachable only on this branch: a wedged POST /api/chat/threads left the user unable to delete that chat or to clear history at all. Both paths now give up after 5s and proceed. That is safe: deleteStoredChatThreads tombstones the ids, so a row landing late is masked from every read and removed by the next delete. * studio: close the gaps around a retried or late thread row write retryFailedThreadRecord consumed the stored creator before starting its retry, so a second caller arriving in that window found no creator, reported the row absent and returned without waiting. updateStoredChatThread then dropped its patch and saveStoredChatMessage raced its PUT against the retrying POST. It now falls through to the pending write when the creator is already claimed. That retry also swallowed its own failure. Returning undefined reads as no row to update, which is how the prompt queue permanently clears shouldCorrectPersistedModel and leaves the thread carrying the live checkpoint instead of the one captured for the queued run. The failure propagates now. A write that outruns the bounded wait still commits, and a local tombstone only hides the row here while the backend keeps serving it. deleteLateThreadRecords issues a second delete once such a write settles, for both the per-id path and clear-all. * studio: retire thread row creators a clear outran, and re-delete until stable A write in flight when history was cleared still ran its rejection handler afterwards, which recorded its creator again after the map had been emptied. The uncommitted thread was in neither inventory so it got no tombstone either, and the next ensureStoredChatThread consumed that creator and recreated a thread the user had cleared. Creators now carry the clear epoch they were registered under and only record themselves while it still matches. deleteLateThreadRecords also snapshotted the tracked promise once, so a creator chained on after that snapshot was missed and the follow-up DELETE could run before the newer write committed. It waits through awaitStoredChatThreadRecord instead, which drains until the entry stops changing. * studio: retire retry creators once a sibling write succeeds, and count point imports Overlapping initializations with mixed outcomes each recorded a retry creator, including the ones whose sibling had already persisted the row. Nothing cleared them on success, so if that row was later removed without this tab taking a tombstone, the next autosave consumed a stale creator and recreated a deleted chat. A resolved tracked write now drops them. importLegacyThread also creates a backend row without moving legacyChatImportGeneration, so a listing that read before a point-lookup import could skip its re-read and, once the import hint was set, drop the legacy copy too, returning a list missing the recovered thread. * studio: await in-flight point imports, and drop document materialization across a clear The point-import generation bump lands only when the POST promise resolves, not when its commit becomes visible to other requests, so a listing holding a stale first read could compare an unmoved generation and skip its re-read. With the import hint set it then drops the legacy copy too, returning a list missing the recovered thread. Listings now await imports already in flight. ThreadDocumentsBar also kept waiting through a clear that had given up on its thread POST. The follow-up delete then raced the upload, and upload_thread_document does not check that the chat row exists, so documents stayed indexed after a reported successful clear. The bar captures the clear boundary and abandons materialization when it moves. * studio: include creators registered during the clear drain in late cleanup The late-cleanup list was snapshotted before the drain, so a chat initialized while the drain was running, whose write also outlived the deadline, was in neither that list nor the backend inventory the clear builds. It took no tombstone and no follow-up delete, and reappeared after a successful clear. The list is now the union of the pre-drain ids and whatever is still pending when the drain gives up. * studio: bound the append's row wait and pin its incognito decision The append's write promise is registered with chatHistoryClearBoundary, and clearAllChats awaits that boundary's unbounded waitForPending before it ever reaches the drain in clearStoredChats. A wedged row write therefore hung Clear all for the life of the request. The append waits through the bounded helper now, so the drain's deadline is the one that applies. ensureThreadRecord also read the incognito toggle live, so a retry after a failed write picked up whatever the toggle said later rather than what the original send captured. A user who enabled incognito in between had an already-initialized normal thread marked temporary, which 404s the save in flight and silently stops persisting the ones after it. initialize() snapshots the toggle and the retry reuses that value. * studio: bound the rest of the append path, and close three follow-up races Bounding only the append's first wait left the hang in place: saveStoredChatMessage still enters ensureStoredChatThread, which waited on the same creator with no deadline, and the append promise is what clearAllChats blocks on through the boundary's waitForPending. That wait is bounded now. Late cleanup captured its ids after the delete returned, by which point the tracker had already dropped a write that settled while the request was in flight, so the follow-up delete was skipped and the late row survived. Both the per-id path and clear-all capture before issuing the delete. The point-import drain snapshotted the set once, so an import starting during the wait was missed and the listing could compare generations before that row committed. It drains until empty. Thread creation checked the project and inserted in separate steps, which the threadpool now runs concurrently with a project delete. The foreign key then failed with an untranslated sqlite3.IntegrityError and a 500; save_thread and patch_thread report the same 404 instead. * studio: bound the retry wait without losing its failure, and 404 a lost fork source retryFailedThreadRecord still waited on the creator with no deadline, so the append could keep clearAllChats blocked through the boundary's waitForPending even after the two earlier waits were bounded. It cannot simply reuse the swallowing helper: this wait has to keep throwing, or updateStoredChatThread reads a stalled write as nothing to update and drops its patch. A bounded wait that rethrows, and reports a timeout as a failure, satisfies both. fork_thread reported a source deleted between its reads and the fork transaction as a 500. The threadpool conversion in this branch is what lets a delete interleave there, so it returns the same 404 the preliminary read would have. * studio: close the windows the bounded waits opened, and 404 a vanished research parent Bounding the retry wait introduced its own gap: after a timeout the caller's GET can 404 just before the write commits and clears the tracker, and the branch that found neither a creator nor a pending write then reported a row that now exists as missing. It re-reads instead. The repeated point-import drain also had no deadline, so one wedged saveChatThread would hang every sidebar, count and search listing rather than just the lookup that started it. It shares the same five-second bound as the other drains. Point backfills write a row without being tracked, so a listing could observe neither the patched row nor its generation bump. They register alongside point imports now. create_research_run reported a thread deleted between its checks and the insert as a 500, the same foreign-key race the sync-handler conversion opened for thread creation and forking. * studio: serialize thread row writes instead of draining around them Ordering between a thread's row write and everything that follows it was enforced by whoever remembered to wait: two registries, a clear epoch, three flavours of bounded wait, a drain-until-stable loop and a capture-then-re-delete pass. Each of those covered the window it was written for and left the next one open, because work registered after any snapshot escaped it. Every row write and delete for a thread now runs on that thread's queue. A delete enqueued after a create cannot overtake it and a create cannot slip past a delete, so ordering holds by construction rather than by draining, and the late-delete follow-ups are gone with it. Waiting is one bounded helper: queued work still runs when the wait expires, so a wedged request delays a delete rather than hanging it or the clear boundary that tracks the send. The clear epoch stays, narrowed to its real job of retiring creators captured before a clear. Net 85 lines removed. * studio: keep a retry re-registerable and let a failed delete surface The queue rewrite dropped two behaviours it should have carried over. retryFailedThreadRecord enqueued the creator directly instead of going through trackStoredChatThreadRecord, so a retry that also failed never re-registered it. Since assistant-ui caches a resolved initialize() and never asks for the row again, two consecutive failures, which a local backend restart produces routinely, retired the thread's only recovery path and every later write 404d for the rest of the session. Its other branch returned undefined without re-reading, so a write committing between the caller's read and that check read as a missing row and updateStoredChatThread dropped its patch silently. deleteStoredChatThreads wrapped the delete in settleWithin, which resolves on rejection as well as on timeout, making the function infallible. The sidebar's optimistic-tombstone rollback and the failed-delete toast both live in catch blocks and had become unreachable, so a rejected DELETE looked like success while the local copy was destroyed. The wait stays bounded; the outcome no longer does. * studio: preserve queued delete failures and wait before titling * studio: close pending thread lifecycle races * studio: close final chat persistence races * studio: preserve retry snapshots and confirmed deletes * studio: close chat deletion races * studio: make chat deletion final across late writes * studio: cancel research before project cleanup * studio: make thread deletion state authoritative * studio: centralize durable chat history coordination Thread initialization resolves before its row write completes, so delete and clear operations must remain final across late or transport-ambiguous writes. SQLite tombstones fence pending thread IDs, clear retries reuse an operation ID and return the exact deleted IDs, and the frontend closes row-write admission while a clear commits. This removes the browser keyed write queue and result classifier. Chat exports read projects, threads, and messages from one SQLite snapshot. Listings preserve fields from still-pending legacy backfills after the bounded wait without restoring legacy-only rows rejected by backend tombstones. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * studio: bound chat thread writes at the request instead of racing timers The bounded-settlement helpers existed because authFetch set no request timeout, so a wedged socket could hang a delete, a clear or a thread listing forever. Racing timers around the promise papered over that: a 5 second bound turned slow-but-successful work into a reported failure, and the duplicate DELETE it then issued contended with the first on sqlite's write lock, so the retry timed out too and the whole operation rejected after the backend had already committed. Pass an AbortSignal timeout on the thread row writes instead, reusing the existing timeoutSignal ponyfill. Every wait then settles on its own and the client-side machinery goes away with bounded-settlement.ts and its test. Follow-on corrections that machinery was hiding: - The abort path no longer rolls its tombstone back when the DELETE rejects. The backend tombstones inside the delete transaction, so a rejection whose request did commit would resurrect an id that then 410s on every later write, silently dropping the conversation. - retryFailedThreadRecord returns undefined for a thread with no stored row rather than throwing. generateTitle, rename, archive and the prompt queue's model correction all treat a missing row as a no-op; the throw turned that into a rejected title stream and a multi-select archive that reported failure for chats it had archived. It also drops the second GET that repeated the caller's own miss. - A clear that fails no longer tombstones threads whose row write was in flight. The epoch bump already retires those creators, and the local tombstone was hiding chats the clear never removed. - clear_chat_history_with_active_research_runs reports only ids whose row existed. Fenced ids for writes that never committed are still tombstoned, but counting them inflated the cleared-chat toast. - Restored the RAG fast path for a thread that already exists, so a transient GET failure no longer drops the upload with a toast. - append() publishes activeThreadId before its point reads again. Gating it on the read skipped temporary chats entirely and left the runtime on the previous chat when the read failed. - listStoredChatThreads waits once on the imports already in flight rather than draining to a 5 second deadline, which one wedged import turned into a permanent stall on every sidebar, search and count. - get_connection passes a 30 second busy timeout. The route handlers run in Starlette's threadpool now, so BEGIN IMMEDIATE writers contend, and sqlite's 5 second default escaped as an unmapped 500 during a large clear. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * studio: keep the longer sqlite wait off the event loop The 30 second busy timeout is only safe for handlers Starlette runs in its threadpool. Every route in research_runs.py was still async def while calling blocking sqlite directly, so db.create_run could hold the event loop in BEGIN IMMEDIATE for the full wait and freeze unrelated requests and streams. The seven handlers that never await are now sync, matching what chat_history.py already does; research_events keeps its async def because it awaits. ensure_chat_project_workspace also had to stop recreating a workspace that a concurrent delete just removed. Now that project reads run in the threadpool they can interleave with DELETE /projects/{id}?delete_files=true, and a read that recreated the directory after the row was gone left an orphan nothing could clean up. It re-checks the row when it actually created the directory and removes it again if the project has since gone. * studio: scope the extended sqlite wait and fence project deletes The 30 second busy timeout was applied to every studio.db connection, so it also reached the routes still running on the event loop. training_history.py:update_training_run is one: it stays async def and calls update_run_display_name directly, so its UPDATE could freeze unrelated requests and streams for the full wait. get_connection is back to sqlite's 5 second default and the longer wait is now passed only by upsert_chat_thread, delete_chat_threads_with_active_ research_runs and clear_chat_history_with_active_research_runs, all of which are reached only from chat_history.py and therefore only from the threadpool. ensure_chat_project_workspace decided whether to re-check the project row from a snapshot taken before the directory was created, so a delete landing during _ensure_project_workspace still left an orphan. It now re-reads the row after the create in every case. upload_project_document had the same gap on the RAG side. Its project check and ingestion.start_ingestion straddle a delete_project that now runs in the threadpool, so an upload could land a document and a stored file scoped to a project whose own RAG cleanup had already run. It re-checks after ingestion and discards the document and its upload when the project has gone. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * studio: stop an ingestion job writing chunks for a deleted document start_ingestion spawns a daemon worker, and chunks carry no foreign key to documents, so a document removed mid-job left the worker free to finish embedding and call store.add_chunks afterwards. The chunk, FTS and vector rows then survived under a scope nothing can reach: delete_chat_project's own RAG cleanup had already listed the project's documents, and the new discard path in upload_project_document could not see them either. The worker now re-reads the document immediately before the write and cancels the job when it has gone, which fences every deletion path rather than only the one that prompted this. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * studio: close the remaining races around thread and document deletion Four gaps left by the previous rounds. The ingestion worker re-read its document before store.add_chunks, but the read and the write were separate statements with no lock between them, so a delete committing in that window still stranded chunk, FTS and vector rows. The recheck now runs inside BEGIN IMMEDIATE, so a concurrent delete_document waits behind the insert instead of slipping between the two. deleteStoredChatThreads treated any rejection as a failed delete. A request aborted by the new 30 second timeout can have committed on the server, and the callers roll their optimistic tombstone back on a throw, which resurrects an id the backend has permanently tombstoned. It now re-reads the rows and only propagates the failure when they really survived. ThreadDocumentsBar skipped its persistence check for an id it already held, but append() publishes activeThreadId before the row read, so that id can belong to a thread whose write failed and upload_thread_document does not check the thread itself. Both paths go through requireStoredThread, which blocks only on a definitive miss so a transient read failure still cannot drop an upload. The chat write timeouts used timeoutSignal, whose ponyfill path for WebViews without AbortSignal.timeout keeps its controller and timer alive for the full 30 seconds. They share threadWriteFetch now, which disposes the signal once the request settles. * studio: finish cancellation and clear contracts * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Drop the unused thread read and route the documents bar through the chat barrel for PR #8136 * Restore failed row-write propagation and guard the discarded-upload cleanup for PR #8136 * Bound the delete reconciliation reads so a wedged socket cannot hang a delete for PR #8136 * Keep a cancelled ingestion job terminal across a restart for PR #8136 * Close the clear timeout, tombstoned-upload and unclaimed-run gaps for PR #8136 * Do not re-signal research runs when a clear is replayed for PR #8136 * Guard the empty-parse completion and stop deleting a kept workspace for PR #8136 * Guard the ingestion completion after add_chunks commits and raise the clear fence bound for PR #8136 * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: shimmyshimmer <michaelhan2050@gmail.com> Co-authored-by: oobabooga <112222186+oobabooga@users.noreply.github.com> Co-authored-by: danielhanchen <danielhanchen@gmail.com>	2026-08-10 03:39:50 -07:00
Daniel Han	ba4f7274a1	Studio: show the files a tool call creates, and keep them in one place (#8256) ... * Studio: show the files a tool call creates, and keep them in one place Files written by the python or terminal tool were unreachable from the chat: only images could be fetched back, nothing could list a chat's files, the terminal tool reported none at all, and the model was never told what happens to what it writes, so it guessed at a temporary directory and users had to search the disk. - both executors report created files via a __FILES__ sentinel, stripped before the model sees it - non-images download as attachments, images still render inline - GET /sandbox/{session_id} lists a chat's files and its path - tool cards get a 'files created' row with working downloads - the sandbox moves under the studio home (migrated, overridable) instead of a third folder in the user's home - UNSLOTH_COMPILE_LOCATION is pinned before unsloth_zoo reads it, so the compiled cache stops landing in the launcher's working directory - deleting a chat cleans up its sandbox * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Do not create a sandbox from a read, and reject reserved names Serving or listing a file resolves the sandbox path instead of creating it, so a GET no longer leaves a folder behind for every id it is asked about. Session ids matching a Windows device name (CON, NUL, COM1) now collapse to _invalid, and the compiled-cache cleanup follows the configured location rather than only the defaults. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Cover the cases the first pass missed Files written into a subdirectory are now found, listed and downloadable, and a run that wrote its file before timing out or being cancelled still reports it. Clearing all chats removes their sandboxes like deleting them one by one does. The legacy migration is serialised and runs for a read too, so an upgraded install does not 404 until some tool happens to run. The image list is capped like the file list, the __FILES__ envelope is only stripped when it parses, and a direct `uvicorn main:app` start pins the compiled-cache location as well. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Replay the tool text, and keep the sentinel honest A file-producing call was replayed to the model as the whole card object rather than the stdout it saw; the same shape was already reaching that path for image results, so both go through their text now. The snapshot walk and the download route share one segment limit, so the card cannot advertise a file the route would refuse, and both ends of the __FILES__ envelope check each entry rather than only that it is a list. Size joins mtime in the change key for volumes with coarse timestamps. * Drop the duplicate import the main merge left behind * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Report exactly what the route will serve The snapshot and the listing apply the download route's own per-segment allowlist, so a name containing a backslash or a control character is never offered as a chip that would then 404. Dotfiles a tool wrote are reported now, since .gitignore is a real artifact and the route serves it; dot-directories stay out, which is where the noise lives. Deleting a chat runs the legacy migration first, so a delete before any tool has run still finds the folder. * Do not delete what is not ours, and do not serve what is not inside UNSLOTH_COMPILE_LOCATION is user-set, and routing the cleanup through it let a value naming a shared directory take that whole tree at startup; a configured path now has to look like a compiled cache before anything deletes from it. The read-only resolver applies the containment check the executing one always had, so a session entry that is a symlink out of the root no longer becomes the root and serves its target. The listing walk drops directory segments the download route would refuse, and executor scratch no longer keeps a deleted chat's folder alive forever. * Unwrap the result everywhere it is read, not just on replay The export paths feed fine-tuning datasets, and they serialised the card wrapper whole; they share one helper with replay now, which also covers the image shape they never unwrapped either. A file the user named studio_exec_results.csv is no longer treated as the executor's scratch, which is specifically studio_exec_<random>.py. And a sandbox root Studio did not create keeps its permissions, since the override can name a shared directory. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Prove the cache is ours before deleting it A directory of plain .py files is somebody's package, so the shape test was still too loose: Studio writes a marker file when it creates the location, and a configured path needs that marker or a generated module name before anything deletes from it. Containment moved into one helper applied to cached session paths too, so a directory swapped for a symlink after it was cached stops resolving there. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Delete only our own files from a shared compile location A directory holding a generated module was treated as ours whole, so a UNSLOTH_COMPILE_LOCATION pointing at a shared directory lost its siblings. Only a directory Studio created (marker) is cleared whole; anywhere else just the generated modules go. Also cap directories visited during the sandbox snapshot, and treat a RecursionError from an oversized __FILES__ payload like the invalid JSON it is. * Bound the listing walk, and retry a legacy move that failed The listing route grew its own copy of the walk and drifted from the snapshot: no directory budget, a prefix match that hid the user's own studio_exec_results.csv, and a hardcoded file cap. It now shares all three with the snapshot walk. Migration flagged itself complete even when a move raised, so a file locked on Windows was stranded once the destination directory appeared. It reports whether anything movable is left, and only then is it done. Clear-all-chats takes the same delete_files opt-in as DELETE /threads, still off by default. * Never delete through a session symlink, and keep the cache marker A session entry replaced by a symlink to a sibling passed the realpath containment check, so deleting that chat took the other one's files. The link itself is dropped and its target left alone. Startup clears the cache it just created, and nothing rewrote the marker afterwards, so the next cleanup demoted our own cache to shared. It is restored after clearing a directory we own. A chat's first turn has no thread id yet but still runs in a real workdir (_default, which the UI and the download route both resolve), so its files get a download chip too. The model- and API-facing message is still stripped, which is the boundary that has to stay clean. * Drop a stray file a local test run left in the tree * Reserve no filename, and prove ownership of a CWD cache The executor's scratch script moves out of the sandbox, so a tool writing studio_exec_results.py keeps it: nothing in the sandbox belongs to Studio except the remap sidecar it writes itself, which is excluded by exact name. That sidecar was being reported as a user file, which also made a streamed result differ from a non-streamed one. A folder named unsloth_compiled_cache in the launch directory is no longer ours by construction; it needs the marker or generated modules like any other configured location. Containment uses commonpath, so a sandbox root that is a filesystem or volume root no longer sends every chat to _invalid. * Put the scratch script back in the sandbox, and gate first-turn files Moving it to the system temp directory changed sys.path[0], so a module an earlier call wrote stopped importing, __file__ pointed outside the sandbox, and the script's directory became world-writable /tmp. It is back in the workdir; the reporting exclusion is now this call's exact filename rather than a pattern reserved over user names. A turn with no chat id runs in the shared _default workdir, so reporting its files pinned a card to a directory the next new chat would overwrite. Gated again until a per-chat sandbox exists. register_compiled_cache_on_path applies the same ownership test as cleanup, so an unrelated folder in the launch directory cannot shadow dependencies. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Do not unlink a sandbox a tool is using, or claim a directory we found Deleting a chat mid-call removed the empty workdir out from under the running tool, and a process whose cwd is gone fails every relative write. Sessions with a call in flight are tracked and refuse removal. mkdir(exist_ok=True) does not mean Studio created the directory, but the marker that followed licensed an rmtree of it. It is written only when this call actually made the directory. * Decide and delete under one lock, and offer chats the same file choice The busy check released the lock before touching the filesystem, so a tool could start in the window and end up running in a directory the same call then removed. The check and the unlink are one critical section. Chat delete now offers the same switch project delete has, so a chat that wrote files is not left with an unreachable folder. Off by default, and the sandbox is only removed when the user asks. * Rename the delete switch state now that chats use it too * Queue a refused delete, and only offer the switch where it works A delete refused because a tool was running was dropped, and the thread was already gone from history, so nothing would ever name that session again. It is queued and performed when the call ends. The delete switch appeared for training runs and for chats inside a project, where nothing it promises happens. It is shown only where a sandbox can actually be removed, and every opener clears it so it cannot arrive preselected from a cancelled dialog. rmtree refuses a symlink and ignore_errors hid it, so a symlinked UNSLOTH_COMPILE_LOCATION was never cleared. Resolved first. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Tighten the comments in the sandbox and cache paths * Prove ownership before following a cache link, and contain the fallbacks Resolving the cache path made rmtree follow a link, so a built-in path that is a symlink would have taken its target with it. Being at a built-in path proves ownership of the directory, not of what a link there points at; only the marker on the target does. _default and _invalid are ordinary directories in a writable sandbox, so one replaced by a symlink became the root every unchecked request read from. They go through the same containment as a session. A file delete now detaches the directory under the lock and removes the tree afterwards, so a large sandbox no longer blocks every tool start or the event loop of the route that asked. * Finish a detached delete a restart interrupted The detached directory sits under a name no session id can reach, so an exit mid-delete stranded it for good, which is the accumulating folder this change is about. Each delete clears the leftovers of earlier ones. * Sweep interrupted deletes at startup, and stop guessing at project ids The delete worker is a daemon, so an exit left the renamed directory with nothing to reclaim it until someone deleted another chat. Startup sweeps them once, off the request path. A session id starting with project- is only a project workspace when the project exists; an imported chat carrying the prefix gets an ordinary directory from _get_workdir and was never cleaned up. Studio's tool wrapper always carries images, so requiring it stops a foreign result with text and sessionId from being reduced to its text on replay and on every export path. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Name the encoding in two test writes * Prove a sandbox is ours before deleting it The root can be an existing shared folder the user pointed us at, so a chat id can name something already in there. Every session directory we create now carries a marker, and outside our own root that marker is what a delete needs. The tombstone sweep matched any name containing .deleting-, which in such a root reached a user's own report.deleting-backup. It now matches only the exact name the rename produces. Two ids differing only in case are one directory on Windows and on a default macOS volume, so the in-flight bookkeeping folds them together while the queued delete still carries the exact id. * Only ever reclaim what Studio itself created The marker was written on every resolve, so a chat id naming a folder that was already in a shared root claimed it. It is written only when the call created the directory, and a tombstone now needs the marker too, so a user's own archive.deleting-<hex> is left alone. Queued deletes are held per exact id under the folded key: on a case-sensitive filesystem two ids that differ only in case are two directories, and the second request used to replace the first. Deletion in a cache directory we do not own is limited to unsloth_compiled_module_*.py. Unsloth*Trainer.py still identifies a cache, since an old install can be left holding nothing else, but it is a name a user's own subclass can carry and is not proof we wrote it. * Leave a shared root's own entries exactly as they are Dropping a _default or _invalid symlink is only ours to do at our own root. Where the user pointed us at a directory of their own, the entry stays and a fresh one is used instead, so no lookup follows it and nothing of theirs is unlinked. A directory that was already there also keeps its permissions: it is not marked, it is not deleted, and now it is not chmodded to 0700 either. * Give a directory one owner, and name it in the marker The marker now records the id the directory was made for. Two ids that differ only in case are one name on Windows and on a default macOS volume, so the second one gets a directory of its own instead of sharing files that either chat's delete would remove, and a delete is refused outright when the marker names someone else. A session entry that is a symlink is only unlinked at our own root; in a directory the user pointed us at it is their entry. The delete switch said anything this chat's tools wrote. A chat moved back to Recents wrote its earlier files into the project workspace, which chat deletion does not touch, so the text now says what it does. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Step around folders that were already in a shared root A tool can write anything into the directory it runs in, so a marker written there is not evidence. The answer is not to run there at all: a directory already sitting in a root the user pointed us at is stepped around, and this chat gets one of its own. Assignment goes through one claim: the directory is created and the marker taken with O_CREAT|O_EXCL under a lock, so two case-variant ids starting at the same moment cannot both take the same name. A sandbox moved up from the legacy root is claimed as part of the move. Without that an overridden root read it as someone else's and the chat could never remove it again. The terminal card now uses the same wrapper test as the adapter, and a filename holding a lone surrogate is not advertised: encodeURIComponent throws on it, so that chip could never download. * Survive a clobbered marker, and finish a move that stopped part way A marker holding anything but an id now reads as no owner rather than as somebody else's, and at our own root the chat takes its directory back. A tool writing over that file used to send its own chat to a fresh sandbox on the next launch, with everything it had made left in the old one. A destination that exists but was never claimed is a move that was interrupted, not a collision: a cross-device move copies and then unlinks, so both sides are left holding files. The rest is moved up on the next launch, and anything already above wins, the same rule whole directories follow. * Claim what we migrate, and answer only for ids a session can have A destination is claimed as soon as it is recognised as this move interrupted, so a file that could not come up no longer leaves the whole directory reading as somebody else's and the chat resolving to an empty one. In a shared root the move goes to the name the session will resolve to, unless the plain name holds nothing but a copy of what is still below, which is this same move rather than a stranger's folder. A sandbox whose claim did not take is never used: a fresh directory is made instead, since running there would put the chat inside someone else's files. The sandbox listing and download routes now refuse an id no session could have used. Everything unusable collapses into one _invalid directory, so answering for those ids handed a caller the files of every other session that had been given one. An inherited UNSLOTH_COMPILE_LOCATION= is treated as unset, rather than pinning the cache to an empty path. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Keep the record of where a chat's files are outside the sandbox Which directory belongs to which session is now written under the studio home, where tools never run, so a marker one of them deletes or writes over no longer sends that chat to a fresh directory with its files left behind. The marker stays as the on-disk claim; the record is what makes it durable, and either one is enough to delete the folder again. Migration provenance comes from that record too. It is written before the move and dropped when it finishes, so an interruption is the only thing that leaves one behind, and a directory that merely holds similar names is no longer mistaken for one of ours. An id the filesystem cannot hold gets a directory derived from it rather than a shared _invalid bucket, so those chats stop reading and deleting each other's files, and their download chips work. The routes serve them again for the same reason. The legacy move runs at startup instead of from the first read: across filesystems it copies every session, which is not something a listing on the event loop can wait for. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Separate the namespaces, and never read a directory that is not the chat's The derived _id- names are reserved: an id that already looks like one is derived too, so a literal id can no longer land on the directory of whichever unusable id hashes to it. Migration bookkeeping moved into its own object for the same reason, where no session id can name it. A read decides on what is already there, so it now checks ownership of the disambiguated name as well and returns a path under a directory we never create when the answer is no. An entry that is a symlink is stepped around whether its target is inside the root or not: claiming through one writes our marker into a directory somebody else made. The record outranks the marker when reclaiming, since all marker contents are writable from the sandbox and a valid-looking overwrite is no different from a deleted one. Chat deletion runs in a worker: right after an upgrade it also runs the legacy move, which is not something the event loop can wait on. The delete switch is offered for a chat in a project too, whose pre-move files are in its own folder. A files value that is not a list is not a wrapper, since the cards map over it. And a compiled cache in the launch directory needs a file only the compiler writes before it goes on sys.path. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Reclaim a sandbox of empty folders, and keep a linked cache usable A tool that only ran mkdir, or deleted what it wrote, left a directory tree with no files in it, which the default delete refused and no chat could reach again. Empty now means no files of the user's. A built-in cache path that is a symlink had its target removed by the clear and nothing put back, so the link dangled and the next compile could not write through it. The listing walks up to a couple of thousand entries and stats each one, so it moved into a worker like the delete paths. A persisted files array is only a wrapper if every entry has a name: the rows read it, and one null took the whole chat view down. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Trust nothing a sandbox can reach without checking it first The marker is written and read without following a link, so one replaced by a symlink no longer truncates whatever it points at, and does not read as an owner either. A cached path gets the checks a fresh resolve makes. A tool can rename its own directory and leave a link to another chat's in its place, and containment alone accepts that. A recorded directory is only used when it could have been ours: inside the root, not a link, named after this session, and not held by another one. The ledger is under the studio home, which a relative path from a sandbox still reaches, so an entry in it must not be able to name anything this chat would not have been given anyway. A recorded move target that names another root is stale: the override can change between the interrupted move and the retry. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Drop the record only once the directory is gone A delete that keeps the files was clearing the note of where they are, which after a tool has taken the marker is the only thing left saying that directory is this session's. It goes now when the removal really happened, and deletion checks ownership through the same trusted-record test the resolver uses. * Lock the ledger between processes, and stop running in borrowed folders Two Studios can share a studio home, so the read-modify-write of the ownership file takes a lock file as well as the in-process one, steals one a dead process left behind, and proceeds rather than losing a write if it cannot get it. A _default or _invalid that was already sitting in a shared root is the user's: a call with no session id used to run straight in it. We make one of our own instead, claim it and remember it, so it is the same folder on the next call rather than a new one each time. A name kept on both sides of an interrupted move is deliberate, so the move is finished with it. It was reading as failure, which left the migration flag unset and rescanned the whole legacy tree on every first tool call. An id a path segment cannot carry now rides in a query parameter: ASGI decodes %2F before it matches a route, so a slash in an API client's id arrived as a different id and a different filename. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Say in the ledger when a sandbox is in use, and stage what is arriving A call in flight is noted in the ledger as well as in memory, so a second Studio sharing this home does not rename a directory its tool is working in. A delete that arrives then is left in the ledger and carried out at the next startup, once nobody holds it. An entry older than a tool call can run is ignored, so a process that died holding one does not keep the folder for good. A cross-filesystem move is assembled under a name nothing resolves and renamed into place, per directory and per file, so a download card opened during the migration cannot read a file that is half copied. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Name it exactly, hold it per process, and finish what was staged A recorded directory has to carry a name this session could have been given, or a marker naming this session. A prefix test let a chat called a name the folder of one called ab. The busy note is one entry per process and keyed the way the in-memory count is, so two Studios in one sandbox both hold it, the first to finish does not clear the other's note, and two ids differing only in case see each other's. A delete handed over by another Studio carries the exact id and is drained when our own call ends, not only at the next startup. A move interrupted between the copy and the rename is put in place on the next launch: the legacy entry is gone by then, so nothing else would ever look at the staging directory again. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: renew sandbox leases, keep every queued delete, unblock the first tool call Busy leases are renewed while a call is still running, so a call with no timeout cannot have its folder deleted out from under it, and the read and write of the busy entry now happen under one interprocess lock. A recovered staging directory is claimed and recorded like an ordinary move, and only directories a migration of ours actually staged are promoted. Queued deletes accumulate per session instead of overwriting each other, a recorded fallback directory survives losing its marker, clearing every chat reports the ids it deleted, and the first tool call migrates only its own folder while the rest of the tree moves in the background. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: never take a link for a marker, and keep a delete from racing a tool call A cache marker reached through a symlink read as proof the directory was ours, and the cleanup deletes such a directory outright, so a link planted in an unowned unsloth_compiled_cache took the user's files with it. Ownership now needs a real file. A legacy session entry that is a symlink is left where it is: the move preserves the link, and the marker written afterwards would land inside whatever it points at, outside both roots. A first tool call migrates only its own folder and starts the rest in the background, rather than waiting out a copy of every session, and the two paths take the same per-entry lock so one session is never moved twice. Deleting a chat now cancels the generations still running for it. The in-flight guard only covers calls already inside the executor, so a request that had not got there yet could recreate the folder afterwards and write files no chat can reach. Session ids are hashed with surrogatepass, since an API client can send a lone surrogate and a POSIX name decoded with surrogateescape carries them too. File cards stream a download to the chosen path instead of buffering it into a Blob and copying that across IPC, which a multi-gigabyte artifact would not survive. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: let a delete run outside the tool lock, and offer files from every surface Deleting a chat held the lock every tool start takes for the whole rmtree, so removing one large sandbox stopped calls in every other chat. The tree is renamed out of the way while locked and removed after. A cross-filesystem move fills the destination as it goes, so a run killed part way left a partial copy that the next launch read as a session the new root already had, stranding the original. It is assembled under a name nothing resolves to and renamed into place, and a failed attempt takes its own staging tree with it. Only the sidebar dialog offers to delete the files, and after a delete from anywhere else the folder is unreachable. The route now reports which sandboxes it kept and the shared delete path offers them, rather than leaving one behind per chat. The snapshot key carries a digest for files small enough to read: on a coarse clock a rewrite of the same length inside one tick matched on mtime and size, and the call reported no file at all. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: take no directory in a shared root that we did not make The fallback name was claimed even when a directory of the user's already sat there, so a chat ran inside their files and a delete with the switch would have removed them. Both names now get the same test, and a fresh one is taken rather than anything already present. A marker reached through a symlink counted as ownership, since isfile follows one, which made an unrelated directory deletable. The path a chat with nothing of its own resolves to now lives outside every sandbox root, so a folder the user keeps at that name is never listed or served. A request-path migration goes through the resolver like the whole-tree pass, rather than stopping at the plain-name collision and leaving the chat with an empty sandbox and its files at the old root. Deleting a project now takes the sandboxes of the chats it held, which are otherwise left with no record pointing at them. Native file cards ask for an absolute loopback URL, which the native download command requires: a relative one was rejected before the request was made. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: find fallback sandboxes by marker, repair clobbered markers, sweep interrupted deletes - a fallback with a name nothing can recompute is found again by its marker, on a later launch, on a read and on a delete - a marker tool code removed from a directory this run claimed is written again rather than the directory being read as somebody else's - an interrupted detached delete is finished on the next launch, ours only - a pre-upgrade chat whose id starts with the derived prefix is migrated from its literal legacy directory name - clear-all reports the sandboxes it kept and offers to delete them, through the same helper the sidebar delete uses * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: fix the Python tool image path test on the new sandbox route The module import needed the .ts extension the node runner resolves, and an id with a path separator now travels in the query rather than in the path. * Studio: keep the file envelope ours, and land every sandbox move somewhere free - the fallback suffix is encoded like the name it is appended to, so an id with a lone surrogate can still step aside - a legacy move picks a free target when both derived names are the user's - a read finds a marked random fallback, not only creation and deletion - a chat started during clear-all is cancelled before its sandbox goes - a call no longer reports another in-flight call's scratch script - a marker line a program printed itself is broken before ours is appended * Studio: report what a delete kept, and never delete the only copy of a move - a removal deferred behind a running tool call is reported as kept - a project delete uses the membership its transaction deleted, cancels the late ones, and returns the member sandboxes it preserved - a staged legacy move that fails its final rename keeps the tree it already moved, marked, and puts it back at the legacy root when it can - closing an incognito chat cleans up the sandbox its tool calls created - a symlinked directory counts as a file of the user's * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: bound the snapshot hashing, and claim only what one call wrote - a snapshot stops hashing after 64 MiB and falls back to mtime and size - a file written while two calls shared a workdir is claimed by neither - a read serves the legacy directory while the background move is unfinished - a markerless sandbox is only deletable by the id its name carries * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: a shared workdir claims nothing, and one sweeper does the deleting - a call that ran alongside another in the same workdir reports no files, with no dependence on filesystem timestamps - a delete migrates its own session rather than the whole legacy tree - the file envelope is read only from the tools that emit one - a deferred removal runs after the global session lock is released - a staged migration that could not be moved in is adopted by the resolver - detached trees go to one sweeper thread instead of one per chat * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: one migration lock per session, and a sentinel path that cannot escape - a chat's legacy copy no longer blocks another chat's first tool call - the empty-sandbox sentinel derives its name, so an absolute session id cannot make it resolve to a directory of the system's - a legacy copy whose destination is already this chat's is left where it is - an interrupted delete at our own root is swept without needing its marker * Studio: derive every fallback name, and delete the project the transaction did - fallback names are derived and recomputable, so a big shared root can no longer hide a chat's folder from the scan - the startup migration picks a free target like the request path does - a staged move is marked before the rename, not after - a delete uses the directory this process created when its marker is gone - the project delete acts only on the membership its transaction deleted - the client reads the file envelope only from the tools that emit one * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: adopt a sandbox by name, and keep what the old shared bucket holds - a directory is only adopted under a name derived from the chat's own id, so a marker a tool rewrote cannot hand one chat another chat's files - the pre-upgrade _invalid bucket is read where it is and never moved - a user's own .unsloth_sandbox file is preserved by the migration - a project workspace is removed after its chats' tool calls are stopped - a sandbox a surviving fork still shows cards for is kept - the second research path names its tool when stripping * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: wait for a project's tools, and match a sandbox reference exactly - the workspace delete waits for the member chats' tool calls to unwind - a reference is a decoded sessionId value, not a substring of the message - a workspace a surviving fork still shows cards for is kept - the project delete cancels the research runs its transaction removed, by id * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: hold a session closed while its sandbox goes, and stop the right tools - a start for the session being removed waits, so no call is handed a folder the removal is about to rename away - the project wait covers project-<id>, which is what its tool calls run as - a kept project workspace still resolves after the row is gone - a marker-named file is preserved unless it is the exact marker being written - clear-all cancels the research runs its transaction removed, by id - the supervisor is signalled even when the run row has already cascaded * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: remember a kept workspace, and never delete one still in use - a workspace whose idle wait timed out is kept, not removed under a live call - a kept workspace is written down, so a custom location still resolves and the next delete collects it once nothing points at it - a marker a tool rewrote with another id is corrected, its file preserved - the empty-sandbox scan runs with the global lock released - a compiled-cache directory needs a generated file, not just the name * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: record a kept workspace either way, and finish the delete it promised - the path is written down whether or not files were to be deleted, since the row that knew it is gone in both cases - the record says whether the user asked for the files, and only those are ever collected - a delete held up by a running tool call finishes when that call ends, and the startup sweep picks up one a kill interrupted * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: hide only the sandbox's own bookkeeping, and keep the records writable - a pending workspace delete is collected after any deletion, not only one that asked for files - a nested file named like the marker is an ordinary file in both walks - the orphan records live under the studio home, which is ours to write - an inline image below a subdirectory keeps its path separators * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: keep the old shared bucket to the ids that shared it - only an id the previous code could not use as a name reads _invalid, so an ordinary chat is never served another chat's files - a markerless directory is this chat's only when its name says so, on the read path as well as the delete path - a workspace delete that failed stays pending instead of being forgotten - a deferred workspace delete removes the whole project workspace, as the immediate one does * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: keep a retry record, fail safe on a locked database, claim the default sandbox * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: tighten the sandbox comments * Studio: serialise a legacy read with its move, retry a detached delete, keep an overwritten marker * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: reserve the fallback names, recheck a shared call, and keep a recreated chat's files * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: validate every recorded delete, pin the download to the file it checked, and reclaim a fork's source * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: a linked root is the user's, and a recreated chat or project keeps its files * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: answer the download probe, and key kept-folder records by kind and id * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: keep a directory this run claimed, list the root once, and unwrap only our own tools * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: keep the sandbox wrapper check a type guard * Studio: recover on startup, keep a reused project id from stranding the old workspace * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: final pass over the sandbox comments * Studio: skip the snapshot digest where mtime already separates the writes --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-08-10 01:11:46 -07:00
Nilay	9c2fda378e	Studio: stop research threads 409-ing on autosave, keep server-managed messages intact (#7956) ... * Studio: keep server record for research messages on thread sync instead of 409 * Keep research messages prune-exempt even when research updates are allowed * Cover the research autosave route end to end for PR #7956 * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Tighten comments for PR #7956 * Reseat a protected message the prune orphans for PR #7956 * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: keep the prune exemption and the reseat walk in agreement pruned was every stored id the payload omitted, but the delete exempts research messages, so omitting a whole research pair walked the report past a prompt that actually survives and detached the turn. Subtract research ids from pruned so the walk only steps over rows that really go. Also realign the conflict tests in test_chat_history_storage.py: the bulk sync keeps the server copy rather than raising, so they now pin that the client's claim is dropped instead of that the batch 409s. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: reseat research rows the batch does not write, not just protected ones allow_research_update empties protected, but the delete exempts research rows either way, so an omitted research row survived pointing at a parent the same batch deleted. Derive the reseat candidates from research_ids instead, minus the ids the batch itself writes so an authorized reparent is not overwritten by the repair. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: danielhanchen <danielhanchen@gmail.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-08-08 02:50:29 -07:00
Maheswar Kumar	d9e033a9b5	studio: fix deep research progress reporting, loopback auth and model retries (#8129) ... * studio: fix deep research progress reporting, loopback auth and model retries Deep Research looked broken from the frontend even when the backend was working. The plan was ready long before the panel showed it, the header sat on "0 sources · 0 actions", and every model call opened the API monitor. Stall: routes/research_runs.research_events waited on db.wait_for_events through the default executor, so open followers starved the supervisor's own to_thread writes. The wait now runs on a dedicated _EVENT_WAIT_EXECUTOR, with db.get_run left on the default executor so a short read never queues behind parked waits. chat-adapter.ts also owned the stream, so a slow reader froze the card; research-run-store.ts owns it now and the adapter reads through watchResearchRun. Progress: phase.started/progress/ended bracket planning, each decision, the audit and synthesis. Plan step titles stream out of the partial JSON. The header shows the running step without an invented denominator, and a completed fetch step no longer claims it found nothing. Loopback auth: auth.storage.is_internal_api_key lets _request_used_api_key exclude Studio's own workflow keys, so research steps stop opening the API monitor. The answer is memoized, since it runs on the event loop per request. Model refusals: each wait takes modelTimeoutSeconds / (_MAX_MODEL_WAITS + 1) so the wall clock no longer buries the real 400; a 404 that names the model is capped at _NAMED_MODEL_WAIT_SECONDS; and 503 model_switch_failed honours Retry-After instead of giving up in three seconds. Search: _web_search separates a rate limit, a timeout and an empty sweep instead of collapsing all three into "Search failed". _research_step_failed treats an empty result with no knowledge-base evidence as a failed step. Autosave: syncExportedRepositoryToBackend echoes the stored copy of a research report and its prompt, so the 409 no longer discards the whole thread payload. parentId stays the client's, or a delete would persist a link to a pruned row. Cleanup: prompts, redaction, citations and parsing move to core/research/, and _completion, research_runs_db.update_step and stopResearchRunFollower are gone. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * studio: let a pruned parent relink a research prompt instead of 409ing Deleting any message above a research turn still dropped the whole autosave. The client relinks the protected prompt to the deleted node's parent, and the guard compared that parentId against the stored one, so sync_chat_messages raised and the route returned 409. The prune set is now computed before the guard runs, and a parentId change on a protected message is accepted only when its stored parent is being deleted by the same request and the new parent is exactly the surviving ancestor the server itself walks to. Content, metadata, attachments, role and createdAt are compared as before, and a relink to any other target is still refused. * studio: keep the new research tests on the declared Python floor Both files annotate with PEP 604 unions in signatures, which Python evaluates at import, so they pushed tests/test_python39_compatibility.py past its ratchet and would TypeError on the 3.9 floor declared in pyproject.toml. * studio: split the agent-action assertions into their own test They were the only reason the surviving scope had to re-point its imports at the extracted modules, which the import-hoist verifier reads as a rename. * studio: clear the api key origin cache alongside the hash cache is_internal_api_key sizes itself against the hash cache, so clearing one without the other lets the origin cache grow past its bound. Deep Research mints a fresh internal key per model call, so it is the workload that reaches it. * studio: assert search failures against the real ddgs exception classes The tests defined their own DDGSException, RatelimitException and TimeoutException, so they passed by construction. ddgs is unpinned and has renamed these before, and the classifier matches on the class name. * studio: surface a failed stored-message read instead of syncing without it Swallowing it sent the unreconciled payload, which the backend rejects wholesale, so a transient read failure came back as a confusing 409 about server-managed messages. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * studio: normalize the surviving-parent walk the way its caller reads parentId The helper returned the stored parent_id verbatim while the guard compares against `message.get("parentId") or None`, so an empty stored parent_id made the two disagree and refused a legitimate relink. A self-link left behind by a corrupt chain now resolves to the root instead of handing back the message's own id. --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: danielhanchen <danielhanchen@gmail.com>	2026-08-07 23:34:38 -07:00
Michael Han	467e564f56	Studio: keep chat titles whole so a wider sidebar shows more of them (#8078) ... * Studio: keep chat titles whole so a wider sidebar shows more of them Chat titles fell back to the first user message cut at 48 characters with a literal "...", so the clip was baked into the stored title and widening the sidebar could never reveal more. Store the whole first line and let the sidebar row clip it with CSS, capped at 120 (the rename maxLength). Titles already stored pre-cut are rewritten from their first message on load, in one batched call. A title has to be the exact old cut of its own message to qualify, so a rename ending in "..." is left alone, and the patch leaves updatedAt alone so Recents keeps its order. * Studio: harden the legacy chat title repair Review follow-ups on the title migration. Cut the fallback title on code points. A UTF-16 cut can halve an emoji, and the lone surrogate that leaves parses fine but raises UnicodeEncodeError when the backend binds it into SQLite, so the write failed for those chats. Read the titles again right before patching, so a rename that lands while the messages request is in flight wins over the rewrite. Keep a thread marked attempted only once it has been read and written. A failed read or PATCH now clears the mark so a later refresh retries instead of leaving the row clipped for the session. Cap each pass at 100 rows and hold writes to 4 in flight, so a long history drains over a few refreshes rather than putting its whole backlog on a synchronous SQLite route at once. * Studio: guard the legacy title rewrite with a conditional PATCH The last review round narrowed the rename race to the patch round trip but could not close it: a read and a write are two requests, so a rename landing between them was still overwritten. Add expectedTitle to PATCH /api/chat/threads/{id}. The guard rides in the UPDATE's WHERE clause, so the check and the write are one statement, and a stale rewrite gets 409 instead of clobbering the user's title. Other callers are unaffected: without the field the patch applies as before. The repair now sends the title it planned from and drops the extra list call the previous commit added, since correctness comes from the guard rather than from narrowing the window. * Studio: keep the title repair moving through a long history Three review follow-ups on the migration. Reuse the messages the sidebar just fetched. listStoredChatThreadsWithMessages already batched every listed thread's messages and threw the map away, so the repair was refetching up to 100 of the same histories on startup. It now hands the map over and the repair only calls out when it has none. Read Dexie when the backend has nothing for a thread. A legacy chat whose import has not landed yet reads empty from a successful batch call, so the catch-based retry never fired and the row sat marked with its title still clipped. Schedule the next page instead of relying on a write. Paging drained on the history update each PATCH fires, so a page that wrote nothing left the rest of the backlog waiting on an unrelated refresh. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: stop a failed repair page starving the rows behind it The paging added last commit had a hole. Failed writes are unmarked so a later refresh can retry them, but the next page was selected off the same thread list, so it drew those rows straight back in: with a failing PATCH the drain re-ran the same page every 500ms and never reached the rest. The page now hands back the rows it did not take and the next one reads from those. Take the earliest user message rather than the first row of the array. A Dexie read arrives in index order, so a chat whose ids do not sort by createdAt could be compared against a later turn and skipped. The local read is also sorted the way the backend lists messages. Keep a row retryable when nothing was found for it. A read that failed was landing in the map as an empty array, which the repair then took as proof the chat had no opening message and wrote the row off for the session. The map now carries only what was actually read, and rows that come back with nothing stay eligible for a later refresh. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: look locally for any title the read could not explain The local read only fired when a thread came back with no messages at all, so a part-imported legacy chat, holding its opening turn in Dexie while later ones are already on the backend, read as complete. The plan then had no opening message to compare against, found nothing to do, and left the row marked with its title still clipped. Any candidate the first plan could not explain now gets the local read, and the two sources are merged by id before planning again. * Studio: hold the repair to one pass, one row, one source of truth Three review follow-ups. Queue the passes. Several sidebars can be mounted at once and each was free to claim its own page and its own four write lanes, so REPAIR_CONCURRENCY only ever bound a single pass. Let a local read fail on its own. A rejecting Dexie read came out of the gather step and threw away the whole page, including repairs already planned from backend data, and returned before scheduling the rest. It now leaves that one row unexplained and the page carries on. Stop trusting a local-only message once the import is done. Deleting a message prunes the backend and leaves the Dexie copy behind, so merging unconditionally could put a deleted opening prompt back and expand the title from it. Local rows are now trusted on the same terms listStoredChatMessages already merges them on: while the import is unfinished, or when the backend holds nothing. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: repair titles from stored messages only The local read is gone. Its rule trusted Dexie whenever the backend held nothing for a thread, and that is also what a chat looks like once every message in it has been deleted: the prune clears the backend and leaves the Dexie rows, so the repair could rewrite a title out of a prompt the user removed. That is the second way this has been wrong, and narrowing the rule again only moves the next case along. A chat whose messages are not stored yet now reads as unknown, which already leaves it retryable, so it gets rewritten on a later pass once its import lands rather than by reading around the backend. The case genuinely lost is an import that never completes, where the title stays clipped. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: repair titles from a read of its own, taken at write time Two ways deleted content could still reach a title. The sidebar's map was not backend-only. Its entries for a thread the batch call returned nothing for come from listStoredChatMessages, whose zero-backend branch merges the Dexie rows, and a chat whose messages have all been deleted looks exactly like that. Removing the repair's own Dexie read last commit closed one door and left this one open. That map is also fetched at sidebar load, not at write time, so a prompt deleted after the load could still be expanded into a title much later: the title guard does not notice, since deleting a message leaves the title alone. The repair now makes its own batched call per page, immediately before planning and writing, and takes nothing from callers. listStoredChatThreadsWithMessages goes back to its original shape. The window is now one pass rather than the whole session, though a delete inside that window is still possible. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: confirm the title on the client too, not only in the guard expectedTitle is enforced by the backend, so it only guards where the backend knows the field. The desktop app ships its own frontend and can meet an older one, which ignores the field and applies the write, and this API layer already carries fallbacks for exactly that pairing. The repair was assuming a guard it might not have. Read the page's titles back immediately before writing and drop any row that no longer holds the title it was planned from. One call for the page, taken last. Where the backend does enforce expectedTitle the write stays atomic; where it does not, this is what respects a rename. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: only run the migration where the guard is enforced, and fit the cap A backend from before expectedTitle drops the field and applies the write, so on that pairing a rename landing between the title read and the PATCH was still lost. Probing by sending a conditional patch is not available: on the backend in question that request is itself the harm. The served OpenAPI schema answers without touching anything, so the migration now runs only when ChatThreadPatch declares the field, and anything unreadable counts as unsupported. A failed probe is not cached, so one hiccup does not park the migration for the session. The cap also overran the rename input. It kept 120 units and then added the ellipsis, and counted code points where maxLength counts UTF-16 units, so an emoji title could reach 241 of them and could not be edited until a character was deleted. The ellipsis now comes out of the same 120-unit budget, still without splitting a pair. * Studio: keep an HTTP hiccup from parking the migration The capability probe cached a false for any non-success response, so a 401 while the token warms up, or a 503 during startup, meant the backend read as unsupported for the rest of the session and no title was ever repaired. Only a schema that arrived and parsed settles the question now; anything else is retried on the next pass. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: guard a repaired title on the prompt it came from Deleting the opening message does not change the thread title, so expectedTitle alone still matches and the repair could expand text the user had just deleted. The write now also carries expectedOpeningMessageId, checked in the same UPDATE, so a prompt deleted between the read and the write answers 409 and leaves the title alone. Both sides pick the earliest user message the same way, breaking a shared timestamp on id. The schema probe looks for both fields. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: pin that a deleted opening prompt is a decided answer A candidate with no stored messages may still be importing, so it is unmarked and retried. One that has messages but no matching opening prompt is a complete answer: the prompt was deleted or edited and no later pass can prove the title. Unmarking those would re-select them on every refresh, since the title stays clipped and keeps matching the pre-filter. * Studio: keep the title migration off the storage layer Three problems on the write path, all from going through updateStoredChatThread and re-listing the history: It ensures the thread first, so a thread deleted on another client was re-imported from the Dexie rows still held here, resurrecting a conversation the user deleted. The PATCH now goes to the backend directly, so a missing row stays missing and answers 404. The pass only runs where the backend enforces the guard, which makes the client-side title revalidation redundant, and it listed every thread the account has once per page of 100. Removed. A chat whose messages were all deleted reads back the same as one still importing, so it was unmarked and re-read on every refresh for the rest of the session. The import ledger tells the two apart, and is only fetched when a page actually has one. * Studio: tighten the chat title migration comments * Studio: drop lone surrogates from a title under the cap too The cut sanitises what it walks, so only over-cap input was safe. A first line already inside the budget was stored as it came, and an unpaired surrogate in it survives JSON.stringify, reaches the backend intact and fails the SQLite bind, so the title write 500s. Dropping now happens on the whole first line, before whitespace is collapsed, so a surrogate removed from between two spaces does not leave the pair behind and one at the end leaves no trailing space. * Studio: tighten the remaining chat title migration comments --------- Co-authored-by: Unsloth <michaelhan@Michaels-MacBook-Pro.local> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: danielhanchen <danielhanchen@gmail.com>	2026-08-07 07:09:37 -07:00
Daniel Han	8b34acebcd	Studio: stop the macOS launcher killing a warming backend, and the Mac tab blackout (#8076) ... * Studio: recover the MLX self-heal when uv cannot resolve the venv interpreter On Apple Silicon the self-heal reinstalls mlx/mlx-lm/mlx-vlm to re-enable Train/Export. It ran uv with --python sys.executable and nothing else, so when uv declined that path the repair gave up and Train stayed disabled for good: MLX self-heal failed (staying chat-only): error: No virtual environment or system Python installation found for path `<studio_home>/unsloth_studio/bin/python`; run `uv venv` to create an environment A venv's bin/python is a symlink into the base interpreter, and macOS breaks that link routinely: a Homebrew or python.org point upgrade moves the target and the venv keeps a dangling symlink. The running process never notices, because it mapped the binary at exec time, so uv is the first thing to fail. Name the environment as well as the interpreter. _uv_python_target prefers sys.executable and falls back to the venv root when the interpreter no longer re-resolves, _mlx_install_env sets VIRTUAL_ENV from sys.prefix, and a uv run that still reports an unresolvable interpreter is retried once against the venv directory. Only that specific failure retries; an ordinary resolution failure stays a single run. VIRTUAL_ENV is set from sys.prefix rather than forwarded from os.environ, matching how UV_OVERRIDE is already handled: it names the install target, so inheriting it would let a caller redirect the install. When both attempts fail the venv itself is broken rather than merely missing MLX. uv's own text says to run `uv venv`, which would build an environment Unsloth does not manage, so the warning now names `unsloth studio update` instead. Reported on macOS 0.1.524-beta. * Studio: stop double-logging exceptions and sqlite-vec spam A macOS diagnostics bundle came back at ~300KB, and roughly half of it was the same tracebacks written twice. LoggingMiddleware logs request_failed with exc_info, which structlog renders as a full traceback inside the JSON "exception" field, then re-raises. Uvicorn logs that same exception again on stderr as "Exception in ASGI application", and the desktop shell mirrors every stderr line into tauri.log individually, so one failure cost about 90 log lines. Mark the exception once request_failed has reported it and filter uvicorn's duplicate on the uvicorn.error logger, the same technique run.py already uses for the startup line. An exception raised above the middleware carries no marker and keeps its traceback, and --verbose restores both copies. The bundle's own failure was a missing sqlite_vec/vec0.dylib, which the import check does not catch: every /api/rag/knowledge-bases poll opened a connection, failed to load the extension and 500ed. rag_db now warns once per process and raises RagExtensionUnavailable (a RuntimeError subclass, so existing handlers are unaffected), and list_knowledge_bases degrades to an empty list for that case only. A locked or corrupt database still surfaces as before. Also quiet the 2xx line for four boot-burst catalog reads (/api/providers/ registry, /api/providers/, /api/models/loras, /api/settings/personalization); 4xx/5xx and every mutation still log. And tauri.log only checked its 5MiB rotation threshold at startup, so a long session grew unbounded: the file logger now writes through a size-tracking handle that rotates in place. On the reported bundle this removes 945 of 2063 lines (89KB of duplicate stderr traceback) plus the 9 sqlite-vec request_failed events (48KB), leaving one warning line. * Studio: stop Mac launches blacking out the Train and Video tabs The platform store seeds chatOnly from the browser user agent, so on every Mac both rows rendered disabled from first paint, visually identical to a measured "this machine cannot do that", until /api/health answered. Since the hardware detection went lazy that reply can take seconds to minutes. Add capabilitiesUnknown() next to isChatOnly(), derived from the fetched flag that already means "a server-reported verdict is stored" (a deferred reply counts as settled: under the torch-warm kill switch nothing else is coming). NavRowDef gains a pending field, folded in by a small import-free resolver both render sites go through, so an unmeasured row stays enabled and reuses the existing spinner column instead of graying out. The root guard lets /studio and /video wait the verdict out rather than one-way redirecting them to /chat, and each page shows its own loading state while it does. Video also gets a real capability answer. There is no Apple path in the video backend, but a healthy Apple Silicon host is not chat-only, so the tab was enabled and would just fail at load. video_capability() mirrors export_capability() and is spliced into GET /api/system and GET /api/system/hardware as additive fields; the page renders a coming-soon panel on an authoritative false, and the sidebar tooltip is now derived from the reason instead of hardcoding "needs an NVIDIA or AMD GPU". Also retry a failed hardware probe in useHardwareInfo: it resolved to the unloaded default with nothing scheduled to run again, which would have left the new Video gate spinning for the session. * Studio: add a macOS tab-capability UI smoke and nav row test hooks Covers the two things reported on Apple Silicon 0.1.524-beta in one live run: Train and Video rendering blacked out for minutes after launch, and the desktop launcher killing the backend about a minute in with "Server stopped unexpectedly". The nav button now carries data-testid="nav-row-<id>" and data-spinner, so the smoke can tell a spinning row from a greyed-out one. Nothing reads them at runtime; resolveNavRowState still owns the behaviour. tests/studio/playwright_mac_tab_capabilities.py drives a live Studio: it samples the Train and Video rows from first paint and fails if either renders disabled while /api/health still reports hardware_detecting, walks Chat, Hub, Images, Train, Video and Export clicking each row and screenshotting the result, and polls /api/liveness and /api/health on a background thread for the whole run. The poll window is 330s by default, deliberately past the launcher's 300s startup grace: the reported crash landed at t+66s, so a backend that dies to the watchdog fails here rather than looking like a slow boot. Redirects away from /studio and /video are allowed only once the verdict is measured. * Desktop: stop the health watchdog killing a backend that is still importing torch The macOS "Server stopped unexpectedly" report was three probe timeouts inside the first 64s of a normal cold start. #7958 fixed the grace period being bypassed; this is the rest of it, on the probe itself. Probe /api/liveness instead of /api/health. Health awaits hardware detection through _await_hardware_detection on purpose, so probing it every 15s bills the watchdog for the warm thread's torch import. Liveness reads module-level caches only, which is what it was added for. Backends older than the route answer 404, so fall back to health, in the same order process::generic_backend_health_ok and desktop_backend_owner::fetch_liveness use, and accept "alive" or "healthy" so a downgrade still validates. Raise the per-probe budget from 2s to 10s. The C-extension imports hold the GIL and the process can go quiet for seconds at a time on a cold start (3735ms measured on /api/health, with a ~27s silence around it). preflight::backend keeps its own 2s: a timeout there dead-ends the launch instead of retrying, and the backend derives _HEALTH_DETECT_BUDGET_S from that number. Stop one early reply ending the startup grace for good. A backend that answers now can still miss the next three probes while it loads a large model, so has_seen_healthy is set only once a reply says the hardware verdict has settled. /api/liveness now carries the same hardware_detecting marker health publishes, plus hardware_detection_deferred when the warm is switched off and nothing will ever settle it; a backend too old to send either reads as settled, which is what the launcher assumed before. Adds the regression test #7958 landed without: the failure policy replayed against the reported timeline, and the probe covered against a stub backend on both routes. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: accept either password env var in the macOS tab smoke The repo's macOS workflow exports STUDIO_OLD_PW; the staging harness exports STUDIO_PW. Reading only the first made the script KeyError before it reached the backend under staging CI. * Studio: make the new liveness and RAG tests hold on a macOS runner Two assumptions that hold on this dev box but not on a bare macos-15 runner: studio_root_id is environment-derived and is legitimately empty on a fresh runner, so the liveness test asserts the key is present (which is what the launcher reads) rather than that it is truthy. python.org macOS builds ship a sqlite3 without enable_load_extension, so the healthy-path RAG test cannot open a connection at all and errored in fixture setup. It skips there. The unavailable-path tests, which are the ones this PR changes behaviour for, still run everywhere. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: drop the MLX self-heal venv-root retry, which cannot work Codex was right that passing the venv root to uv does not recover a dangling bin/python. Confirmed against uv directly, with a venv whose interpreter symlink was pointed at a missing target: --python <venv>/bin/python error: No virtual environment or system Python installation found for path ... --python <venv> error: No virtual environment or system Python installation found for directory ... VIRTUAL_ENV set, no --python error: Failed to inspect Python interpreter from active virtual environment All three fail the same way, so the fallback and the retry were a placebo: when the first attempt already resolved to the venv root the retry was skipped, and when it ran it repeated a call that cannot succeed. Keep the part that does work. uv refusing the environment means the venv is broken rather than merely missing MLX, and nothing this process can pass to an install command fixes that, so the warning names 'unsloth studio update' (which rebuilds the environment) instead of uv's own 'uv venv' suggestion, which would build one Unsloth does not manage. _venv_root stays: it names the environment in that message. * Studio: hold the startup grace for the whole warm, not just hardware detection The health watchdog ends its five minute startup grace once a liveness reply says the backend is no longer warming, and that signal was `hardware_detecting`. But hardware detection is only the first of utils/torch_warmup.py's stages: the marker disappears while inference_backend, transformers, datasets and unsloth_zoo are still importing, and those are the C-extension imports that hold the GIL longest. So the grace could end mid warm and a stall spanning three 10s probes would count as three dead probes against a backend that was starting normally, which is the unresponsive_health_check kill the grace exists to prevent. Publish `torch_warm_in_progress` on /api/liveness and /api/health, derived from warm_status(), and read that in the launcher. A new field rather than a wider `hardware_detecting`: that marker also means "this hardware verdict is provisional, re-read it", and config/hardware-verdict.ts keeps the UI provisional and polling while it is set, so keeping it lit through datasets would hide Train for the whole warm over a verdict that settled seconds in. Additive and backwards compatible both ways. The launcher keeps its `hardware_detecting` path as a fallback, so a backend that predates the new field still gets the grace it gets today, and a backend that sends it talks to an older launcher exactly as before. The deferred case is preserved by construction: the field is published only while a warm thread is actually running, so it is absent both when the warm finished and when none is coming. UNSLOTH_STUDIO_DISABLE_TORCH_WARM=1 never starts one, and a warm retired mid stage by a shutdown never sets finished; deriving the field from "not finished" would have reported either as warming forever and held the grace open until it expired on its own. * Studio: poll the hardware verdict out of its unknown state on every platform Train and Video now spin instead of graying out while the verdict is unmeasured, so something has to end the spin. fetchDeviceType spends its bounded wait at most once per page load, so a host that detects slower than that keeps the provisional reply and `fetched` stays false. The sidebar's recovery poll returned early unless the host was chat-only or deferred, which off macOS it is not: the store seeds chatOnly from the user agent, so on Linux and Windows nothing re-read /api/health. A cold GPU host importing torch sits squarely in that window, and there the rows spun and /studio held its loading panel until the user navigated or reloaded. Poll while the verdict is unknown too, on any platform, and keep the mlx_unavailable and deferred cases as they were. The poll re-reads with force, which is what gets past the cached provisional reply and the spent wait latch, and the effect re-runs when the verdict lands, so the interval is cleared as soon as it is known. Re-reads are skipped while one is outstanding, bounded so a request that never settles cannot hold the poll off. studio-page and video-page gate on the same store value, and the sidebar is mounted on both routes, so they recover with it rather than growing a second poll. Covered by tests/hardware-verdict-recovery.test.ts, which drives the real store through a slow non-Mac detection and asserts the guard arms on the unknown state. * Studio: teach the run-module test stub about the new loggers export run.py imports install_uvicorn_duplicate_exception_filter at module scope, and load_studio_run_module replaces 'loggers' with a stub module carrying only get_logger, so importing run raised ImportError before any test body ran: ImportError: cannot import name 'install_uvicorn_duplicate_exception_filter' from 'loggers' (unknown location) A no-op is the right stub here. The filter only de-duplicates uvicorn's copy of a traceback and this module never starts a server. * Studio: run the tab-capability smoke in the macOS UI job Codex was right that the script was dead code: nothing under .github invoked playwright_mac_tab_capabilities.py, so the regression coverage it claims never executed and the tab blackout could come back unnoticed. It gets its own boot on a cold port. The assertion is that Train and Video spin rather than grey out while the verdict is unmeasured, and that window only exists on a backend that has not warmed yet, so reusing the already-warm 18897 server would have passed vacuously. For the same reason this phase deliberately does not wait for a healthy backend first, unlike every other phase here: waiting is how you miss the window. The script does its own wait, and health answers provisionally inside a 1s budget. The liveness poll is cut from its 330s default to 120s here. The full window exists to outlive the launcher's 300s startup grace, and nothing in this job runs that watchdog; it boots 'unsloth studio' directly. Spending five macOS-runner minutes to prove something this job cannot observe is not worth it, and the watchdog is covered by the Rust tests in commands.rs. * Studio: let /video reach its own gate, and stop stale polls freeing the guard Two review findings on this branch. /video was still outside CHAT_ONLY_ALLOWED, so on a measured chat-only host, a CPU-only box or a Mac without usable MLX, a direct link or a reload bounced to /chat before VideoPage could render. That is exactly where the unsupported explanation this branch added has something to say, so the message was unreachable in the only cases it was written for. Video now follows /export: the route is allowed through and the page self-gates on the backend's video verdict, so nothing loads on a host that cannot run it. The recovery poll's no-stacking guard could be freed by a read that no longer held it. A read outliving the 30s stall window is abandoned and the next tick starts a replacement, but the abandoned read's finally still zeroed the shared marker, so every following tick saw a free guard and fired another forced /api/health. On the backend this poll exists for, one still importing torch, that is a read every three seconds piled onto the process being waited for. A generation counter now means only the owning read can clear it. Both regressions verified to fail without the fix: 3 of 754 tests go red. * Studio: make the tab-capability smoke fail instead of passing vacuously The first staging run went green having tested nothing. The log tells the story: [mac-tabs] spinner observed during warm: {'train': False, 'video': False} [mac-tabs] Train: redirected to http://127.0.0.1:8888/login ... (allowed) [mac-tabs] Train: nav row not pinned inline; reached by route instead [mac-tabs] PASS The login form takes a username as well as a password and the script filled only the password, so the submit was a no-op and the browser stayed signed out. Every later check then read an empty shell: no sidebar, so no nav row, so no assertion had anything to act on. The redirect check called the bounce to /login 'allowed' because it only asked whether the verdict was measured, and the row checks downgraded a total miss to an info line. Three changes, all so this run would have gone red: log_in fills the username, then proves the session took by navigating to /chat and checking it stayed; a failure there aborts rather than continuing into checks that cannot mean anything. A bounce to /login, /onboarding or /change-password during the tab walk is a failure. The session was proven live before the walk started, so losing it mid-walk is never an allowed redirect. A walk that never locates a single nav row now fails. That is the shape a signed-out or unrendered run takes, and it has to be loud. The survival poll was the one part that did mean something: 65 samples on each of /api/liveness and /api/health, zero non-200, backend alive past 319s. * Studio: make RAG unavailability one coherent state across the router Degrading the KB list to an empty response when sqlite-vec's native library cannot load stopped the 500-plus-traceback on every poll, but it left the rest of the router behind: the frontend read the empty list as a working empty state, offered Create, and the POST went straight to get_connection() and raised. So the fix for the log spam reintroduced the log spam one click later, and told the user nothing. One contract now. The polled KB list still degrades, but carries ragAvailable and ragUnavailableReason so a client can tell "no knowledge bases yet" from "RAG cannot run on this machine". Every other endpoint answers 503 stating the same reason, via a rag_available() gate up front and a _rag_connection() wrapper that catches the case where the first request of a session is the one that discovers the library is missing. That wrapper also reaches the connections ingestion opens for itself, and removes an upload it had already saved rather than orphaning it in the uploads root. The two new fields are additive and the document listings are deliberately left erroring rather than degrading, so a frontend that has not learned to read the marker keeps every error surface it has today. rag_available() remembers only that the extension loaded, never that it failed, so a one-off cannot latch RAG off until restart. The warn-once stays: a 503 path that logged per request would be the same regression in a new place. Genuine database errors are untouched and still surface as real errors. reconcile_orphaned_ingestion_jobs() gates on rag_available() too, so it is the no-op its docstring already claimed instead of raising out of startup to be logged as a reconcile failure. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: let the UI read the RAG availability the backend already reports routes/rag.py answers a host where sqlite-vec will not load as a contract: the polled KB list degrades to 200 with ragAvailable/ragUnavailableReason beside an empty list, and every other endpoint answers 503 with the same reason. The client dropped both, so a broken-RAG Mac still showed an apparently working, empty Knowledge bases page whose Create button could only 503. features/rag/api/rag-availability holds the verdict, modelled on the platform store in config/env: optimistic until the backend has actually answered, so a slow first poll, an unreachable server and a backend that predates the marker all render exactly as they do today. Only a measured unavailable gates anything. Wired in at every response path, including the two that bypass ragRequest (ragUpload, streamJobEvents), so a user who lands on a mutating route first gets a coherent UI instead of waiting for the list poll. listKnowledgeBases reads the marker, which is the only way to tell an empty store from a host where RAG cannot run. The dialog then disables New knowledge base and Create/Save, guards submitForm for the keyboard path, and shows the backend's reason in place of "No knowledge bases yet.". Also breaks a hang this condition triggers. targetHasIndexingDocuments answers "still indexing" whenever its listThreadDocuments probe throws, and dispatchQueuedPrompt reschedules on that with no cap, so on a broken-RAG host a queued prompt on a thread using documents was never dispatched at all. A 503 is now distinguishable, and there are no documents to wait for, so it sends. A transient failure still holds the prompt back as before. This is deliberately not folded into useRagToolDisabled: that is a model capability gate and is false when no model is loaded. * Studio: report video as macOS-unsupported on Intel Macs too video_capability() keyed the macOS branch on is_apple_silicon(), so an Intel Mac fell through to pytorch_not_installed or no_accelerator and was told to install PyTorch or add a GPU. Neither enables video: the diffusers pipelines have no supported macOS path at all, so both Macs get the same honest answer. The accompanying AST test also asserted is_apple_silicon() was named in the function, which kept passing on the word surviving in a comment. Strip comments before asserting so it tests the gate rather than the prose. * Studio: only the backend's own 503 detail is a RAG capability verdict A 503 is what a reverse proxy, Cloudflare or a briefly overloaded server returns, and those bodies say nothing about sqlite-vec. Recording one as unavailable gated the Knowledge bases dialog for the rest of the session behind a transient outage, explaining it with an extension failure that never happened -- and only a 2xx from a gated endpoint could clear it. Match the RAG router's own wording instead. A bodyless or unrelated 503 now leaves availability unknown, which is what the store was built to represent. * Studio: fix the macOS tab smoke signing in, which made every assertion vacuous The helper read the password field with count() straight after domcontentloaded. auth-form.tsx returns null while the auth-status request is in flight, so there is no form in the DOM yet, count() does not wait, and the run fell through to 'assuming desktop auth' and checked an empty signed-out shell. It also filled a username the login form does not have and clicked a button labelled 'Sign in' when the label is 'Login'. Wait for #password, click Login, and wait for the post-auth route. Verified against a live Studio: the helper now reports 'signed in' where it previously did not. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: make the tab-capability smoke create the window it asserts on Second hole of the same shape in this file. assert_row_never_greyed_while_unmeasured computed seen_spinner and only logged it, so when the verdict settled before the browser arrived the loop broke on its first iteration, nothing was sampled, and the run went green having observed nothing. Requiring the window would not have fixed it, because the window is not there to be required. hardware_detecting covers stage 0 of the warm, and on the macOS runner's --no-torch install with no MLX that stage is a failed `import torch` plus one failed metadata lookup: it settles inside a second of the port binding, while Playwright is still launching Chromium. Add the login and two navigations, one of which spends the frontend's own 5s wait on the verdict, and the sampler arrives 15 to 20s late on every host, fast or slow. The workflow comment claiming a cold boot keeps the window open was wrong; it is corrected to say what boot ordering actually buys, which is a provisional reply to the first /api/health probe and nothing past it. So the script opens its own window instead of racing that one. It answers the browser's /api/health with a real reply that has the measurement taken back out, holds it there, and requires nav-row-train to render enabled with data-spinner="true", the way pending beats disabled in resolveNavRowState. That window is open for as long as the check needs, on any host, and a missing row fails rather than skips, so there is no path through it that reports success without having read the row. The real warm is still sampled and a real grey-out still fails, but nothing is required of it. Two more things in here could observe nothing: The Video half of this file was structurally empty. Video is not pinned inline (SIDEBAR_NAV_DEFAULT_PINNED), so it renders inside the More dropdown, which mounts nothing until it is opened and carries no data-testid even then. Every query for nav-row-video returned null on every host, so seen_spinner["video"] could never be true and "Video: nav row not pinned inline" was a permanent info line, not a finding. Train carries the same pending flag through the same resolver, so it is the observable end of that wire; the rows the sidebar does pin are now named, an inline row that does not render is a failure, and the More rows are documented as expected misses. _saw_any_row was satisfied by any row on any route, and the sampler swallowed a dead page with a bare `except Exception: break`. Now the walk requires every default-pinned row to have been seen, and a page that cannot be evaluated at all fails. tests/studio/test_mac_tab_capability_warm_window.py drives all of this with the page and the backend stubbed, so the red cases are checked in the tests/ walk rather than only on a macOS runner. Against the previous file, its first case reproduces the staging log exactly: "spinner observed during warm: {'train': False, 'video': False}", then PASS. * Studio: give an adopted backend the startup grace when it is still warming The watchdog already knows that one healthy answer is proof of life and not proof that startup is over, but only on the path where this app spawned the backend. An adopted backend starts with the latch already set, on the reasoning that it was serving before the app attached. That is the same fallacy: a force-quit during a cold start leaves the backend running and still importing the ML stack, and the relaunched app adopts it. Three GIL-stalled probes later it was cleared and the user got a crash screen for a host that was starting normally. The ownership probe carries no warm-up signal, so ask the backend directly once ownership is verified, and clear the latch on a warming reply rather than merely declining to set it. The grace stays bounded at 300s either way. This path predates the fix on the owned side; it is the other half of the same report rather than a regression. * Studio: fix five guards in this PR's own tests that could not fail Each was verified by mutation: the regression the test names was applied, the suite stayed green, and it now goes red. - provisional-hardware-verdict: the slice end was located by searching for the latch it then asserted was absent. String.search returns the first match, so moving the latch into the loop moved the boundary with it and the assertion passed against exactly the regression its header describes. Anchor on the loop's closing brace, and require the latch to exist after it so deleting it outright is not a pass either. - liveness warm state: the subprocess harness builds its result with body.get(studio_root_id), so the key is present whether or not the reply carried it. Deleting the field from liveness_check() left the test green. Report presence explicitly, as the two neighbouring fields already do. - warm window, DEVICE guard: asserted a count of at least two when the function has three comparisons, so deleting the poll loop's requirement still left two. Bind to the sites instead of counting them. - warm window, boot silence: scanned only for _fetch_top_models, but the fetch is started through the public wrapper now, so putting _start_top_models_fetch back in the constructor restored the huggingface.co call on every boot undetected. Check both names. - rag availability: asserted only after noteRagAvailability, which writes unavailable unconditionally, so the exemption under test could be deleted. Assert between the two calls. Also replaces a dead 'disabled: chatOnly;' string check: that is an object entry, so the regressed form ends in a comma and the literal could never match. * Studio: deliver the hardware verdict to a component that subscribed a tick late useHardwareInfo seeds its state from the module cache during render, but joins the listener set in the effect. A probe resolving between those two points notifies the listeners registered at the time, which does not include this one, and leaves the cache set, so 'if (!cached) load()' skipped the fetch as redundant. Nothing remained that would ever call setInfo, and the component sat on the unloaded default for its whole life. That was survivable while callers read individual fields. This PR gates whole pages on 'loaded', so it now reads as 'Checking this machine for video support' for the rest of the session, which is the stuck-loading state the PR exists to remove. Hand the cache straight to the listener instead. Also stops a successful 200 that a later invalidate superseded from resolving as the unloaded default, which load() reads as a failed probe. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: complete the forced password change in the macOS tab smoke The repo's own macOS smoke hands this script the raw bootstrap password, and a backend that still holds one injects it into the page and signs itself in, landing on /change-password with no login form ever rendered. This script treats that as a signed-out route, since it has no sidebar to assert against, so the run failed with 'could not sign in'. The staging harness rotates over the API before driving, which is why the same script passed there and failed here. Rotate it in the browser instead of giving up, and check for that screen before waiting on the login field so an authenticated session does not spend a minute waiting for a form that is correctly absent. Verified against a live Studio on both paths: a fresh instance holding its bootstrap password now reports 'password rotated' then 'signed in' where it previously reported 'could not sign in', and an already-rotated instance still signs in through the login form. * Studio: give the adopted ownership probe the watchdog's probe budget The warm-up read added for adopted backends is gated on ownership verifying, and that probe runs every request at the 2s default. During the multi-second GIL stall the watchdog exists to ride out, both requests inside it time out, the backend comes back unverified, and the warm-up read never runs at all, so the grace never reopens and three stalls still clear the backend. The longer budget has to be applied before verification, not after it. probe_owned_backend_state keeps its signature and delegates to a variant taking an explicit budget, so only the watchdog changes. A guard binds that call site to HEALTH_PROBE_TIMEOUT and fails if it drifts back to the default. Also uploads the tab-capability phase's log and Playwright evidence, which the artifact list did not name, so a red macOS run discarded the only record of it. * Studio: match the RAG 503 on the extension name only Matching either fragment meant the loose half carried the same weight as the specific one: anything RAG-aware in front of the backend can answer a transient 503 saying RAG is unavailable without meaning the extension, and that persisted a capability verdict which only a later successful gated request could clear. Keep sqlite-vec and drop the English phrase. Nothing upstream emits a package name by accident, and the capability being gated is exactly that extension, so this is narrower than requiring both fragments would be while still tolerating a reworded backend detail. Requiring both would have made the matcher brittle to precisely the rewording the fragment match exists for. Two proxy phrasings added to the generic-503 case, which restoring the loose marker now fails. * Studio: stop the MLX install env claiming a recovery it does not perform _mlx_install_env's docstring still described VIRTUAL_ENV as the second half of a dangling-symlink recovery, and named the helper that performed the first half. That helper was deleted earlier in this PR when the retry was found not to work, but the claim outlived it and is false: an explicit --python outranks VIRTUAL_ENV, so uv reports the same unresolved-interpreter error either way. It is not a harmless stale comment. It is the tree asserting a repair that does not happen, and a reviewer reading it asked for the mechanism to be restored. State what uv actually does, and why --target and --prefix are not the escape hatch they look like: both exit 0 against a broken venv but resolve against whatever ambient interpreter uv finds, writing a wrong-ABI or off-sys.path install that leaves mlx_stack_available() False while looking like success. Two guards: the deleted helper must stay deleted and the claim must not come back, and the unresolved-interpreter path must stay one attempt plus a diagnosis, never a second install with a different target. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: make the watchdog-budget guard line-ending agnostic include_str! embeds the file exactly as checked out, so on Windows the source is CRLF and the \n}\n search for the end of check_watchdog_health never matched. The guard panicked on the Tauri CI runner while passing on every Linux and macOS job. Normalise before searching. Verified both ways: 149 tests pass on the LF tree, and the guard still passes after converting the file to CRLF, which reproduced the runner failure before this. --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-08-07 02:45:06 -07:00
Eyera	b52d3b56ed	feat(studio): rework train page setup flow (#7633) ... * feat(studio): rework training setup and cache handling * Rework train page resource selection * Improve train page workflows and resource selection * Fix train page picker behavior * Fix stale training errors after token updates * Fix training eval steps regression * Fix shared token use when resuming training * Harden train page resource workflows Unify model and dataset picker behavior, training start guards, and token handling. Validate cache provenance for local models, downloaded resources, and processed datasets. Prevent stale preview and history state while keeping the train page modular. * Fix training method race and test isolation * Restore training upload limit exports * Fix training config formatting contract * fix(studio): harden training model selection and preflight * fix training resume, cache, and lifecycle reliability * Fix training stop watchdog path stub * Fix training start state and train page UI * Unify train page selection controls * Fix train page validation, localization, and paging * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Clean up train locales and restore recipes link * fix(studio): correct training model selection and preflight Preserve freeform local model paths, reject remote GGUF-only repositories, and correct the training start snapshot contract. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix(studio): harden training resource selection Preserve local dataset path intent, localize setup-change errors, and remove unused picker tour hooks. Refresh model picker and PDF recipe contracts for the refactored training flow. * fix(studio): remove train tab scrolling * fix(studio): harden training resource preflight Reject missing local models and binary adapter artifacts. Preserve selected model and dataset cache pins through preflight and QLoRA loading. Detect cache path changes before training starts. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix train page validation and cache handling Keep transient model checks nonblocking and bind cached models and datasets to their selected snapshots. Improve Hub auth errors, retries, task search, streaming consistency, locale formatting, reset handling, payload mapping, partial inventory filtering, theme accents, and regression coverage. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix(studio): stabilize train page pickers Unify Hub validation, local inventory, picker styling, and dataset names. Localize training feedback and explain streaming modality changes. Add unit, contract, and cross-browser picker coverage. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix(studio): align cached scans and variant state purge Scan selected cached snapshots during training security preflight. Purge manifests and cancel markers using their stored GGUF variant. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix(studio): harden train cache, consent, and UI Fix cache cleanup and cross-platform model path handling. Align resume and adapter consent scans with pinned load targets. Localize dataset flows and refine navigation, state cleanup, and theming. * fix(studio): harden training resource selection Preserve cached model configuration HTTP errors so stale cache references return their intended 404 responses. Wait for device inventory settlement before locking inferred picker tabs, while keeping known device items visible during scans and retries. Apply modality name heuristics only to the final model path component across supported platforms. Expose full model and dataset identities on truncated picker triggers. Add regression coverage for picker settlement, retry behavior, and modality inference. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix train picker flow and resume safety Scan the exact cached model snapshot before resume consent and preserve the actual repository identity. Keep cold pickers on Device until inventory settles, cap automatic Hub pagination, and provide a stable Load more action. Improve train control and history card accessibility, remove duplicate token labeling and dead styling, and correct picker spacing. Normalize Windows relative model paths and add coverage for resume pins, picker policy, pagination, path identity, and accessibility contracts. Fix the reported formatting and import ordering issues. * Fix cross-platform train dataset path detection Use the shared local path detector so persisted Windows, Unix, UNC, and home-relative dataset references do not render Hugging Face-only controls. Recognize Arrow dataset files and add regression coverage for supported local path formats and Hub repository identifiers. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix training start recovery and dataset selection Reconcile failed fresh and resumed training starts with backend status before reporting an error. Treat the selected dataset source as authoritative so filename-like Hub repositories retain subset and split controls. Align dataset picker sorting and remove unused advisor template state with a persisted-state migration. Add regression coverage for transport recovery and Hub dataset selection. * Fix train picker validation and start recovery Update the resume training contract test for the asynchronous failure handler. Filter invalid Hugging Face search results and block invalid saved model and dataset selections. Restrict local dataset selection to inventoried application paths. Load training YAML through a bounded native file picker while keeping the browser fallback. Only recover uncertain training starts after network or response parsing failures, and match the active job to its request identifier. Treat cached full precision models as ready for QLoRA without showing a false download warning. Add regression coverage for the picker, config import, and training recovery behavior. * Harden training starts and config imports Track training start request IDs from preflight through pending, accepted, and rejected states so retries remain idempotent. Disable automatic retries for start posts and reconcile ambiguous client responses with server request status. Clear terminal training status on reset so later runs do not inherit stale state. Enforce the 1 MiB YAML limit for browser imports and keep read errors specific to the selected file label. Remove unused dataset picker translation entries across all locales. Add focused tests for request idempotency, start recovery, config limits, and picker contracts. * Harden train picker feedback and start handling Show explicit empty states for device dataset searches and invalid Hub model or dataset queries while keeping pagination sentinels mounted. Run the train picker Playwright suite across Chromium, Firefox, and WebKit in Windows CI. Keep training start outcomes owned by backend spawn completion so cancelled handlers cannot mark active jobs as rejected. Correct the Train section heading hierarchy and format the branch-owned native file dialog test. Add picker contracts, browser assertions, and cancellation race coverage for the updated behavior. * Fix training setup races and configuration state Keep hardware-based method selection within model-default loading so starts cannot race changing defaults. Clear start errors only after deliberate configuration or token edits while background reconciliation stays silent. Generate training request IDs safely for LAN HTTP access before acquiring the runtime lease. Filter persisted state, sanitize invalid methods, and guard preview metadata. Localize config size failures, use the toast wrapper, and remove obsolete training files. Add regression coverage for the updated start, persistence, and import behavior. * Refine training configuration and picker workflows Split the training configuration store and parameter panel into focused persistence, policy, LoRA, hyperparameter, memory, and MLX modules. Run picker coverage on platform-appropriate browser engines and parameterize the unmanaged dataset path. Surface configuration changes when streaming is disabled and clear stale dataset modality state after failed probes. Consolidate dataset format and AI-assisted mapping on hub routes with header-based Hugging Face tokens. Hold local path actions until model inventory loading settles. * Fix train picker state and preflight regressions Open online pickers on the Hub before inventory settles and keep the inferred tab stable for the session. Preserve parameter and VRAM metadata for pinned selected models. Reject failed backend starts once and cancel automatic modality corrections without showing a destructive setup error. Update persistence contracts to follow the extracted training configuration module. Move shared parameter option styling out of the React component module to preserve Fast Refresh. Add regression coverage for picker tab behavior, pinned model metadata, persistence contracts, and start rejection handling. * fix(train): harden model selection and start feedback Preserve cache metadata and model capabilities when freeform local paths resolve to discovered models. Return stable training start error codes through direct and recovered requests, then localize Hugging Face preflight failures across every supported locale. Show whether submitted advanced settings are default or non-default in Simple mode, and migrate the parameter mode preference to the standard storage key. Restore absent-key migration semantics and update the embedding security test for the extracted gate helper. Add focused coverage for structured errors, advanced setting summaries, start recovery, and security gates. * fix(train): harden picker validation and focus Align Hub ID validation with Hugging Face rules while accepting valid underscore boundaries. Reject persisted path-shaped datasets from Hugging Face-specific controls. Reuse model identity normalization for cache path comparisons. Move picker search focus into the popover focus lifecycle. Update source contract tests for extracted model selection and structured resume errors. * fix(train): refine picker behavior and dataset structure Wait for device inventory before locking inferred picker tabs while preserving explicit user choices. Align Hub resource ID validation with backend rules for leading and trailing underscores. Support ArrowDown navigation from picker tabs into available options. Split dataset selection, uploads, streaming settings, and inventory refresh logic into focused modules. * fix(studio): stabilize train picker contracts and state Update dataset contracts to follow the extracted selection, upload, and inventory modules. Keep the inferred picker tab stable while device inventory settles. Use the shared TrainingMethod type for VRAM estimation and remove redundant casts. * fix(train): align Hugging Face repo validation Allow underscores at repository segment boundaries to match Hugging Face repo ID rules. Keep model and dataset search queries unvalidated while preserving selection safeguards. Add regression coverage for accepted IDs and unrestricted picker searches. * Fix training resource compatibility and cache safety Restore legacy dataset format and mapping routes while moving training uploads and checks to the canonical Hub endpoints. Enforce configured upload limits, clean partial files after failures or cancellation, and support both upload route families in middleware. Align Hugging Face repository validation with accepted underscore boundaries and skip PyTorch dependent tests when the dependency is unavailable. Keep model format and cache metadata correct when the same repository is selected from another source. Only promote complete runnable model weights as cached training resources, align download notices with picker capabilities, and reject weightless local snapshots during start preflight. Cancel stale dataset mapping assistant requests before they can overwrite a newer selection. Preserve touch selection for training methods by separating tooltip toggling from Select item activation. Add focused backend, frontend, and picker regression coverage for these paths. * fix(studio): finish train page and onboarding rework Ignore cache-only fields when comparing training start inputs so reconciliation does not cancel valid runs. Route native dataset drops through Tauri path validation and lease-backed managed uploads. Use the production model and dataset selectors in onboarding and replace placeholder uploads with real imports. Require positive learning rates while keeping learning rate, epoch, and step inputs editable. Normalize model identities, improve dataset summary order, and guard persisted selection state. Recognize Windows rooted and drive-relative paths consistently in picker logic. Add MLX optimizer help and learning rate validation messages to every supported locale. Add frontend, backend, and native policy regression coverage for the updated behavior. * chore(ci): remove train picker smoke workflow changes Remove the train picker Playwright step from the macOS Studio smoke checks. Remove the train picker Playwright step from the Linux Studio smoke checks. Remove the train picker Playwright step from the Windows Studio smoke checks. Drop the matching artifact upload paths so the workflow files match upstream main. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix(studio): harden train picker behavior and boundaries Expose dataset drop helpers through the training public API and move local path detection into shared code. Block freeform Hub choices while offline for pointer and keyboard selection. Make native dataset drop handling safe for committed React renders. Keep recipe navigation working when session storage is unavailable. Preserve Windows drive root identities and cover the distinction from drive relative paths. Update picker and training input contracts to match the extracted implementations. Remove orphaned translations and align the train style block with the existing indentation. * Fix train picker cache and upload behavior Keep selected model cache paths strict and preserve fallback discovery only when no path is supplied. Offload multipart writes and native dataset copies from the backend event loop. Prewarm train picker inventories so device-first tab inference can settle before the first open. Use shared picker tab constants across the model and dataset selectors. Update picker contracts and add regression coverage for cache resolution and dataset uploads. * Refine train picker controls and component structure Split picker retry, error, and pagination UI into focused components. Add a shared segmented control and use native radio semantics for dataset source selection. Centralize segmented indicator positioning with RTL-aware transforms. Separate dataset advanced settings state wiring from its presentation and source toggle. Make picker arrow navigation move symmetrically from tabs through search and options. * Fix train page reconciliation and picker contracts Preserve tuned training parameters when model cache references change, while refreshing defaults when no user edits occurred. Keep streaming modality corrections visible through the Start CTA after preflight stops. Localize the new training parameter and dataset mapping text across all supported locales. Refresh picker structural contracts after the component split and apply Biome formatting to the changed files. * Fix train upload tests and desktop drop handling Pass an explicit empty native path lease in legacy upload limit tests so direct route calls match FastAPI request behavior. Track Tauri scale factor changes during dataset drag and drop, clean up both listeners safely, and cover runtime updates. Replace the run preview card's important border utility with an explicit dark theme card modifier. * fix(studio): harden training setup and history state Restore Hugging Face token entry and validation in the onboarding model and dataset steps. Apply the configured upload size preflight before onboarding dataset uploads. Update moved model and dataset cache references without showing false missing cache warnings. Parse SSE frame delimiters by their matched length so mixed line endings preserve event data. Match sensitive pid paths by exact segment while retaining database state protections. Clear deleted Data Recipe selections after inventory settlement while preserving direct uploads. Show the files deleted status only when a run previously recorded an output directory. Handle Data Recipe navigation failures and keep the native drop listener stable during uploads. Replace formatting-specific contract checks and add focused behavioral regression coverage. * Harden training start and model picker flows Split the train model picker into its own entry point to keep it out of shared route bundles. Preserve pending idempotent start reservations and reconcile them before accepting a run. Keep unconfirmed starts in a polling state with accurate localized feedback. Continue starts after automatically disabling unsupported multimodal dataset streaming. Localize the updated model picker controls across supported languages. Lock dataset uploads before asynchronous preflight to prevent concurrent selections. Add regression coverage for reservation states and unconfirmed runtime starts. * fix(studio): harden training picker edge cases Preserve POSIX model path identity while retaining Windows relative path normalization, and clarify the override migration behavior. Localize the onboarding model and dataset controls across every supported locale. Truncate native dataset filenames by Unicode code point so generated labels cannot contain lone surrogates. Cancel training after preflight disables streaming for image or audio datasets so users can review the changed setting before restarting. Update runtime contract assertions for the setStartPending rename and add regression coverage for path identity and Unicode truncation. * fix(studio): align training picker controls Restore full-height Browse and Amazon S3 selection with an accessible radiogroup that avoids fieldset sizing behavior. Align model and dataset picker tabs with the shared 36 px segmented control geometry and typography. Remove active borders and focus rings from picker search inputs while preserving keyboard focus indicators on other controls. Add contract coverage for segmented control sizing and picker search focus styling. * fix(studio): harden onboarding picker flows Classify native dataset drops from the full path before shortening display names. Keep onboarding model choices within the selected training type across Hub, device, and freeform selections. Retry model default loading when onboarding restores an interrupted model selection. Localize the Hugging Face token field across supported languages. Reset picker result scrolling when switching between Device and Hugging Face tabs. Add regression coverage for native filenames, model constraints, hydration, and picker scrolling. * fix(studio): reconcile multimodal streaming before training Disable streaming when dataset checks detect image or audio data, then recheck cached selections in non-streaming mode. Keep start-time modality detection as a safe fallback that updates the configuration and continues without an error or a second click. Build onboarding training method options from the shared method order and metadata so onboarding stays aligned with the train page. Add contract coverage for the streaming reconciliation and shared training method list. * Fix train picker contracts and dataset recovery Allow local models without reliable modality metadata to pass onboarding constraints while retaining explicit mismatch checks. Reconcile GPU selection without effect-driven state updates and mark intentional deep Hub imports for lint. Return and consume stable local dataset cache miss codes, and send the canonical train_split field. Move train-specific picker code into its owning feature and update the related contract coverage. Use the canonical Hub dataset progress route from Chat. * Harden training resource selection and offline handling Return an actionable preflight error when offline mode is enabled and the selected model is not cached. Validate evaluation dataset uploads with the shared extension policy and reuse the centralized accept list. Disable streaming for explicit on-device dataset selections while preserving Hugging Face selection intent. Keep dataset identifiers safety checked without blocking benign values before Hugging Face handles repository validation. Remove the branch-added source assertions and cover the new selection policy with behavioral tests. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix training setup validation and async state races Validate Hugging Face dataset IDs and repository access before training starts while keeping local paths on their filesystem validation flow. Scope model default race protection to fields that defaults can overwrite so unrelated dataset edits no longer suppress model initialization. Stop Hugging Face token edits from invalidating token-independent device inventory scans on every keystroke. Add regression coverage for dataset source validation, cached and remote Hub preflight, model default application, and token inventory behavior. * Tidy up train page layout, tooltips and dark mode Layout - Move the Browse / Amazon S3 toggle into the Dataset section header - Put Subset, Train Split and Evaluation Split on one row - Put Target Format, Train Split Start and Train Split End on one row - Pair Project Name with Max Steps, and Context Length with Learning Rate - Centre the hyperparameter tabs and give them a fixed width so the sliding indicator lines up with its segment - Give parameter rows a min height so slider and select rows share one vertical rhythm - Slightly more spacing below section headings and at card bottoms Tooltips - Add hints for Model, Method, Dataset and Project Name - Move field descriptions into the tooltips and drop the inline copy - Upload field keeps the accepted file types inline, with size limit and Learning Recipes note in the tooltip HF token - Show a masked preview of a saved token instead of a generic label - Read Not set when no token is stored Dark mode - Drop borders on the LoRA option cards and target module chips, and separate states with background fill instead - Use a lighter, less saturated green for the selected state Other - Expand LoRA Settings and Training Hyperparameters when Advanced opens - Shorten the upload label to Drop file or click to upload - Localise the new wizard tooltips across all locales * Fix training defaults, MLX validation, and dataset preflight Prevent late hardware recommendations from replacing configuration values loaded after model selection. Block unsupported MLX methods and embedding runs in the UI and backend before training is queued. Require live Hub access for streaming dataset preflight instead of accepting unrelated cached data. Compare advanced settings against applied model defaults and cover the updated behavior with focused tests. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix training picker validation and platform safeguards Update the Tauri YAML contract test to verify save filter behavior without depending on the Rust vector implementation. Align frontend Hub resource validation and training provenance normalization with the shared repository ID rules, including valid underscore and maximum length identifiers. Reuse one frontend training method policy so Apple Silicon onboarding and the Train page consistently disable CPT and prevent unsupported onboarding completion. Add regression coverage for rejected Hub IDs and exact resume provenance. * Fix MLX training capability validation Detect hardware before platform validation so unsupported MLX configurations fail before a job is queued. Reject audio dataset training during server preflight while preserving the worker guard. Share Apple Silicon capability checks across model selection, onboarding, readiness, submission, and LoRA controls. Cover audio, CPT, embedding, LoftQ, DoRA, and warm-start detection with regression tests. * Fix training quit protection, translations, and preference reset Protect desktop quits while a training start is pending or a run is active, and keep the warning text accurate. Localize dataset upload details across every supported locale and require those keys in parity checks. Centralize the training picker and parameter mode storage keys so resetting local preferences clears current and legacy values. Add regression coverage for training activity transitions and preference reset ownership. * Preserve multimodal training support Keep upstream training eligibility authoritative while preserving separate vision, audio, and embedding capabilities. Classify dual audio and vision models for vision training without changing inference support or enabling pure audio training on MLX. Add coverage for model type constraints, backend type resolution, and MLX validation. Apply the pending Biome formatting fixes to the dataset selector and training wizard. * Fix training defaults retry and clean stale translations Allow model defaults retries after failed cache reconciliation without overwriting user-edited settings. Ignore stale fallback vision checks once a newer defaults request starts. Remove unused parameter description keys from every locale and cover the retry behavior in the training contract tests. * Document direct model identity imports Explain why the model identity helpers bypass the Hub barrel beside each lint suppression. * Preserve training settings across reloads Persist applied model defaults and their advanced settings baseline while refreshing transient model metadata without overwriting tuned hyperparameters. Keep remote format probe tests runnable without an installed huggingface_hub package by supplying a fake module. Normalize tilde-prefixed path separators while preserving case-sensitive identities. Remove unmatched model and dataset tour anchors. Add regression coverage for persistence migration and model identity normalization. * Harden train page validation and persistence Validate CPT embedding learning rates against backend bounds and add localized client feedback. Keep invalid learning rate drafts out of persisted state and YAML exports. Notify users when multimodal model selection restores a non-S3 dataset source. Remove W&B tokens from training config persistence and migrate stored secrets safely. Update the model defaults contract test to cover metadata refresh without requiring the removed early return. Move segmented control styles into a non-component module to preserve Fast Refresh behavior. Add validation and persistence coverage for the new contracts. * Fix training model IDs, cache state, and localization Preserve root-level Hugging Face model IDs instead of rewriting them into the Unsloth namespace. Debounce token-scoped inventory reconciliation so token and inventory updates use stable request keys. Keep the selected training history run when the Studio view remounts. Localize dataset subset and split selectors across every supported locale. Complete Italian picker and training translations and enforce Studio-wide locale parity. * Fix training history errors and stale stop state Return structured artifact deletion errors and show localized messages that distinguish active training output from filesystem failures while keeping history intact. Clear superseded stop requests only when they belong to the current runtime generation, and cover start invalidation and stale stop handling with regression tests. Clarify that shared picker styles apply outside the Hub while Hub-only rules remain scoped. * Fix training method persistence and settings summaries Persist training method provenance so manual learning rates, model adapter rates, and pre-CPT dataset formats survive reloads and method changes. Deduplicate imported target modules and compare advanced-setting arrays by value counts so duplicate entries cannot hide non-default settings. Add migration and regression coverage for legacy state, rehydrated method transitions, restored learning rates, and duplicate target modules. * Fix persisted completion defaults and inventory retries Persist trainOnCompletions across reloads so model defaults and advanced settings summaries remain accurate. Defer missing legacy values until model metadata loads, preserving tuned settings while respecting streaming, raw text, CPT, embedding models, and explicit user changes. Settle partial inventory failures when usable rows exist so manual local model paths and Enter submission remain available. Add retry actions for partially failed model and dataset scans, including when filtering leaves no visible results. Add regression coverage for persistence migration, constrained completion defaults, inventory settlement, and picker empty states. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix training validation and route state regressions Retry newly detected image datasets with vision-aware validation, clear stale dataset check failures, and cover the failed background classification case. Restore Train history cleanup and centralize the learning recipes navigation intent key. Repair backend route stubs for canonical model IDs and path normalization, and isolate consent tests from shared package import state. * Preserve training cache pins during preflight Keep model and dataset cache flags and local paths in the pending start comparison so reconciliation cannot submit a stale snapshot. Cover changed cached copies and uncached transitions with focused regression assertions. * Preserve Hub identity for cached training exports Keep exact cached snapshots for training while restoring the standard Hub repository identity before PEFT saves. Recover repository identity in memory for legacy adapters so imatrix exports work without rewriting adapter configs. Cover repository mismatches, local models, and Windows cache paths with focused regressions. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix cached training, evaluation, and MLX resume behavior Limit cached dataset slice loads so small training ranges do not materialize full duplicate caches. Restore Hub model identity for cached Torch and MLX runs so saved adapters remain portable. Attest MLX model provenance across runtime and prequantized 4-bit formats so valid checkpoints can resume. Recognize missing cached eval splits, reload train and eval together, treat explicit eval split errors as fatal, and use deterministic held-out data for automatic fallback. Retain non-fatal evaluation warnings in training status and render them in the train UI. Apply pin and model format validation to every MLX worker entry point and keep the worker test fixture aligned with its imports. * Stabilize training progress updates across job handoffs Keep active and pending training ownership separate so status polling cannot switch identities during a running job. Scope metrics, progress streams, stop, and reset operations to the expected job and recheck ownership during handoffs. Make polling, stream parsing, and start reconciliation abortable, monotonic, and safe against stale overlapping requests. Preserve same-run interface state so live updates do not remount progress controls or dismiss the stop confirmation. Add backend and frontend regression coverage for ownership changes, stale events, request races, and stream cleanup. * Fix cached training fallback and dataset reconciliation Recognize incomplete SentencePiece tokenizer caches and retry online model loads through the Hub while preserving strict offline and resume pins. Prevent rejected dataset snapshots from being promoted again until the selection or inventory changes. Forward cancellation signals to dataset format requests so superseded checks stop at the transport layer. Add focused backend and frontend regression coverage for these cache paths. * Harden offline training and cache fallback behavior Validate cached model snapshots for tokenizer and processor support, then fall back to the pinned Hub revision when the cache is incomplete. Remember rejected dataset cache entries and cancel superseded checks so metadata-only snapshots cannot trigger request loops. Reject deleted local datasets before any Hub access and clear only the matching stale selection in background, preview, and start flows. Reserve training starts before validation so overlapping requests and GPU consumers cannot race the spawn window. Keep GGUF fallback metadata tied to the cache that supplied variants when Hub access fails. Filter DSpark and DFlash drafter companions across picker, loading, inventory, and deletion paths while preserving MTP behavior. * Harden training setup and picker behavior Keep training starts blocked while a stop request is pending and preserve the stop latch through backend failures. Skip malformed training progress events while validating SSE payloads and releasing stream readers safely. Enforce non-streaming state for upload and S3 datasets across source changes, persistence, and rehydration. Route Markdown dataset drops to Data Recipes across browser and native path formats. Give model and dataset source tablists localized accessible names. Refresh training start contracts and add focused regression coverage for the corrected behavior. * Fix offline dataset cache selection and token status Prefer finalized processed dataset caches for offline training while keeping raw cache paths available for scoped management. Treat malformed Hugging Face tokens as unset in Train and surface local validation feedback without unnecessary requests. Add regression coverage for cache coexistence, interrupted cache builds, path propagation, and malformed token presentation. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Preserve sidecar Hub version during dataset cache checks Keep processed dataset cache discovery free of eager datasets imports so training does not preload the base Hugging Face Hub package. Preserve cache root discovery through HF_DATASETS_CACHE, HF_HOME, and XDG_CACHE_HOME before Transformers sidecar activation. * Add the AGPL-3.0 header to the two new studio training contract tests * Scan the fallback target and pin cached snapshots that hold weights Cache fallback scanned the pinned snapshot it was about to discard, then loaded the Hub repo unscanned. Scan after the pin is dropped instead, in all four training paths. Preflight probed the literal model name, so the registry bicodec alias Spark-TTS-0.5B/LLM 404'd and rejected a supported model. Probe the repo the trainer downloads and treat its load subdir as the weight root. An unpinned start still downloads its dataset, so a stray cached copy no longer skips Hub verification. Offline keeps accepting the cache. refs/main can point at a metadata-only revision, so the model pin now prefers a snapshot that carries weights before falling back. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Let the capability-cache guard see through the key type alias model_config.py names the cache key _CapabilityCacheKey, so matching the literal Dict[Tuple no longer finds it and the guard fails on a cache that is still correctly tuple-keyed. Resolve module-level aliases first. Checked by mutation: reverting a cache to Dict[str, ...] is still caught. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: point the extra-UI tour anchor check at studio-model-picker The train page rework renamed the model tour anchor from studio-model to studio-model-picker and updated tour/steps/base-model.tsx accordingly, but tests/studio/playwright_extra_ui.py still looked for the old anchor, so the Chat UI Tests job failed with "[data-tour='studio-model'] not found". Verified against two live Studio instances: the anchor is studio-model on main and studio-model-picker at this head; studio-dataset and studio-params are unchanged on both. * Studio: require metadata and weights together when pinning a model snapshot Pass 1 of _resolve_model_snapshot matched on weights alone, so two cases still selected a snapshot that /training/start then rejects with "does not contain trainable weights": - a newer weights-only fetch (interrupted download, or an allow_patterns pull that never took config.json) displaced an older complete sibling. That is a regression against the previous metadata-first ordering, which started the run; reproduced with a two-snapshot cache where the complete one is older. - consolidated.safetensors counted as weights for selection but is absent from _MODEL_WEIGHT_CANDIDATES in routes/training.py, and transformers 4.57.6 has no loader path for it (zero references in the package), so a config plus consolidated snapshot won over a loadable sibling. Pass 1 now demands metadata AND weights via a required_groups argument on latest_snapshot_from_cache_path, and consolidated.safetensors is dropped from the selection tuple so it matches what the route accepts. Pass 2 keeps the metadata-only fallback unchanged, so caches that never held weights resolve exactly as before. Both new tests in test_model_cache_snapshot.py fail without this change and pass with it; 401 tests across the cache, preflight, provenance and identity suites pass. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: make the new cache tests pass on Windows Four of the PR's new tests fail on windows-latest while passing on ubuntu and macos. All four are defects in the tests, not in the production code. The three blob-symlink tests hardcode a POSIX relative target ("../../blobs/x"). Windows stores the reparse-point substitute name verbatim and resolves it in the object namespace, where / is not a separator, so the link is created but dangles: Path.resolve(strict=True) raises and the provenance and dataset-cache helpers correctly return None. huggingface_hub builds these targets with os.path.relpath (file_download.py _create_symlink), which yields ..\..\blobs\x on Windows, so a real cache never has this shape and no Windows user is affected. Building the target with os.path.relpath / os.path.join matches what huggingface_hub writes. The cross-snapshot rejection tests had the same POSIX targets, so on Windows they were passing for the wrong reason (dangling link rather than the escape check). They now use native separators too, so the rejection logic is actually exercised there. test_runtime_4bit_resume_reaches_worker_with_source_resource_pins compared model_local_path against str(Path); the route posix-normalizes that field via normalize_path, so the assertion now uses Path.as_posix(), which is a no-op on POSIX. 324 tests across the three files pass on Linux. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: bound the cached dataset re-check and stop adopting unconfirmed starts Two defects introduced by this PR's new modules. 1. Unbounded dataset re-check (#7853). training-config-store.ts re-runs the cached format check whenever rejectValidation() reports the in-flight token as stale, with no counter and no backoff. DatasetCacheRejectionTracker advances its generation on any inventory-fingerprint change, and the fingerprint includes sizeBytes, so a dataset that is still downloading invalidates every check in flight and the pair never converges. Measured at 480 requests in 60s; a probe driving the real tracker ran 500 iterations without settling while a stable inventory settles in 1. The retry now draws from a per-selection budget (dataset-recheck-budget.ts) and falls through to the uncached check once spent, so the answer still refreshes when the inventory genuinely changes but a churning one cannot spin. The budget resets when the dataset or split changes. Fixing the tracker fingerprint instead was rejected: existing tests pin the behaviour that a sizeBytes change makes a rejected cache retryable. 2. Unconfirmed starts adopted as recovered. reconcileTrainingStartTransportFailure ended with adoptAcceptedTrainingStart(pending.jobId, ...) and returned "recovered". The backend reserves the request id and job id before the heavy preflight, so a start still pending when the 30s window closes may yet be rejected. Adopting there reported success and pinned a job id that never became current_job_id: acknowledgeTrainingStartRequest was never sent, and dismissTrainingRun's expectedJobId check then returned "superseded", leaving the rejected state unclearable until a new start re-reserved it. It now calls the module's existing settleUnconfirmedTrainingStart and returns "unknown", which both callers already handle by warning startUnconfirmed and settling unconfirmed, and which preserves start_request_id for acknowledgement. New tests fail without each change and pass with it; removing the bound makes them fail rather than hang. typecheck, 598 frontend tests, build, and the 154 studio contract tests all pass. * Studio: retry tokenizer-less pinned snapshots, and skip Hub preflight when offline Two more defects introduced by this rework. Neither mechanism exists on main: worker.py there has 0 occurrences of local_files_only / model_snapshot_path / cache_artifact, and routes/training.py has 0 of model_info / dataset_info. 1. Tokenizer-less pinned snapshot is terminal (#7845). The pin only requires config.json plus a weights file, so a snapshot with no tokenizer pins clean, local_files_only is set, and AutoTokenizer then fails. Recovery is gated on _is_model_cache_artifact_error, whose marker list matches only three of the message shapes transformers actually emits. Sweeping all 159 tokenizer classes against a tokenizer-less snapshot: 37 failures were classified not retryable, of which 26 are genuine cache problems that got zero Hub retry, including XLMRoberta (BGE-M3, multilingual-e5, LaBSE), MBart, NLLB, Bloom, GPTNeoX, Cohere, Marian and the generic PreTrainedTokenizerFast. SentencePiece and BPE families resolve a missing vocab path to None and then dereference it, so the failure arrives as a bare AttributeError with no cache-specific text. Adding those four shapes takes the sweep from 37 misses to 11, and all 11 remaining are correctly fatal (missing optional Python dependency, or an unsupported tokenizer class), which no Hub retry can fix. Widening the classifier rather than validating tokenizers before pinning: the vocab filename space is as open-ended as the exception space across 159 classes, requiring a tokenizer would break the deliberate adapter_config.json pin path, and a real loadability check means loading a tokenizer inside the start request. The recovery path is already gated on offline and require_exact, so a false positive costs one Hub attempt while the current false negative fails the run. 2. Blocking Hub preflight with no reachability guard. Both preflight legs retry metadata at 5s then 10s with no reachability check, and requests applies the timeout per resolved address. Measured 30.0s added to a single POST /training/start with the Hub black-holed, scaling with the number of addresses, surfacing as 503. utils.utils already provides a bounded, memoised hf_unreachable/hf_dns_dead that the training worker subprocess uses one layer down; the route that spawns it did not consult it. The model leg raises inside the existing try, so the except HTTPException handler still runs _resolve_model_snapshot and a cached snapshot pins exactly as before, just without first burning the remote budget. The guard fails open, so an online start is unchanged. New tests fail without each change and pass with it, including a wiring contract that fails if the guard stops being consulted. 443 tests across the preflight, cached-start, provenance, snapshot and streaming suites pass. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Make the import-hoist lint honour __all__ re-exports Source lint failed on studio/backend/utils/security/__init__.py: [BLOCKER] HOISTED-IMPORT-UNUSED 'load_scan_target' (['from:utils.security.file_security:load_scan_target']) added but unused That file is a barrel __init__: every name it imports carries "# noqa: F401" and is listed in __all__, and routes/training.py imports load_scan_target from the package rather than the submodule, so the re-export is load-bearing. HOISTED-IMPORT-UNUSED only excluded re-exports that already existed before the change, so adding any new name to an existing barrel was an automatic blocker. The script's own docstring already records this shape as a known false positive for NEW-UNUSED-IMPORT. Skip module-level imports whose bound name appears in __all__, since those are loaded by importers rather than by the module itself. The rule keeps its teeth: a newly added import whose name is not in __all__ still blocks, and a dangling alias still trips UNRESOLVED-NEW. Verified with a four-case mutation matrix plus --self-test, and the full 79-file changed-set lint now reports OVERALL: PASS. * Keep studio off evaluated PEP 604 unions on the 3.9 floor tests/test_python39_compatibility.py failed on Core (HF=4.57.6, default and latest) and Repo tests (CPU): test_no_pep604_unions_are_evaluated_on_the_declared_floor model_config.py:937: Tuple[...] | Tuple[...] (type alias) test_studio_evaluated_unions_do_not_grow 36 studio files now evaluate PEP 604 unions, up from 35 Both offenders came from this branch. model_config.py:937 is a module-level type alias, so it is evaluated at import time and "from __future__ import annotations" cannot defer it; it needs typing.Union, which is what the test message prescribes. hub/utils/dataset_cache.py is the one file that pushed the count to 36, and all five of its unions are function annotations, so the future import is enough there. No behaviour change: the alias is only a cache-key type, and the annotations are unevaluated either way. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Key the dataset re-check budget on the whole selection The budget added for #7853 keyed on dataset + split only, but a dataset cache usability identity has four user-chosen dimensions: dataset, subset, split and streaming. Changing subset or toggling streaming therefore kept the same key, so a genuinely different selection inherited an exhausted budget, skipped the cache-preferring re-check and dropped straight to a remote resolution with datasetKnownCached cleared. Key on all four instead, JSON-encoded so no delimiter can collide with a name containing it and null stays distinguishable from the string "null". cachePath is deliberately still excluded even though the usability identity carries it. It is derived state that moves as a download populates the cache, so feeding it into the key would mint a fresh budget on every poll and re-arm the exact non-terminating loop this module exists to bound. Tests: two new cases cover subset and streaming, and an inverse case pins that nothing outside the selection can refresh the budget. Reverting the key to dataset::split fails exactly the two new cases and no others. * Stop the pinned snapshot path reaching PEFT as the base model name A completed run wrote a machine-local path as the adapter's base model: base_model_name_or_path = '/home/user/.cache/huggingface/hub/ models--unsloth--Llama-3.2-1B-Instruct/snapshots/0123…' where main writes the Hub id. It lands in adapter_config.json, every checkpoint-*/adapter_config.json, the run card, export_metadata.json for merged and GGUF exports, and the model card push_to_hub uploads, none of which resolve on another machine. restore_hf_cache_repo_identity runs in UnslothTrainer.load_model before get_peft_model, so its peft_config branch has nothing to repair yet, and PEFT then derives the name itself: # peft/mapping_func.py new_name = model.__dict__.get("name_or_path", None) peft_config.base_model_name_or_path = new_name PreTrainedModel.__init__ copies config.name_or_path onto the instance, so restoring only config._name_or_path leaves that slot holding the snapshot path. Restore the instance attribute too. It goes through the same guard as the other fields, so an ordinary local model, an unrelated Hub id and a repo mismatch are all still left alone. Tests are behavioural: the existing model identity suite asserts the call site via AST and stays green with this bug present, which is why it was missed. Removing the new line fails 3 of the 6 added tests and none of the other 11. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Resolve cached snapshots that load from a subdirectory unsloth/Spark-TTS-0.5B keeps everything trainable under LLM/: its snapshot root carries only README.md and config.yaml, no config.json and no weights. _resolve_model_snapshot only looked at the root, so a cached copy resolved to None. _apply_model_cache_pin then warned "Cached copy not found on disk; downloading", and offline the start route turned the same None into a 409 hf_model_not_cached_offline for a model that was sitting in the cache. The remote preflight already handles this: it expands load roots through load_scan_target. Only the cached path disagreed. Reuse security_load_subdirs, which already reports ("LLM",) for BiCodec, so both paths share one source of truth, and apply it to the resume and preflight pin lookups too rather than just the fresh-start one. Detection can raise offline or for a gated repo, so a failure degrades to root-only rather than propagating. Tests build the real models--org--name/snapshots/<rev> layout. Making the helper a no-op fails 3 of the 6, and a snapshot with nothing loadable in either place still resolves to None, so the widening cannot mask an empty cache. * Use Optional in the deprecated dataset alias signatures The rewritten alias module annotates two parameters as UploadFile | None and str | None with no postponed annotations, so they evaluate at import on the declared 3.9 floor while the rest of the file already uses Optional[...]. The file was an evaluated-union offender before this change too, so this is not a new break, just keeping the new code consistent with its own convention and off the debt list. * Apply the repo kwarg-spacing formatter to the new code pre-commit.ci flagged the ruff-format-with-kwargs hook on this branch. Running scripts/run_ruff_format.py locally keeps the fix in the authoring commits instead of trailing a separate bot commit. * Say why a resume is refused instead of blaming the checkpoint can_resume_run gained a provenance clause in this rework, so it now returns False for runs whose checkpoint is entirely intact, most realistically once the pinned model snapshot is evicted from the HF cache. The start route answered every False with "Resume checkpoint must belong to a stopped or errored run with complete saved trainer state." pointing the user at trainer state that is fine, while the real cause was the resource gate. exact_resume_resource_requirements already raises with a precise explanation and resource_provenance_allows_resume was discarding it. Add resource_provenance_resume_blocker, which returns that explanation (or None when the run is resumable), define allows_resume in terms of it so the two cannot drift, and use it in the start route when it is the actual cause. The gate itself is unchanged: refusing an unattested resume is deliberate, only the diagnosis was wrong. Tests are behavioural, with one narrow wiring contract for the branch that regressed. Replacing the precise reason with a generic string fails the message test and nothing else. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Honour load subdirs everywhere a cached snapshot is probed Follow-up to 590ac9f22. Resolving cached Spark-TTS/BiCodec snapshots from their LLM/ load root got a start past _resolve_model_snapshot, but three later probes still asked for root-level config.json only, so the same snapshot was accepted in one place and rejected in the next: - provenance.py attested incomplete model provenance, after which exact_resume_resource_requirements refused resume for a snapshot still sitting on disk; - the start preflight reported a valid cached model as having no trainable weights; - the worker cleared model_snapshot_path before loading, falling back to the Hub instead of the selected snapshot. Promote the helper to hub.utils.hf_cache_state.with_load_subdirs so there is one definition rather than four, and use it at every site. _has_trainable_local_weights takes subdirectory roots instead, since it probes directories rather than a filename list. Unchanged for ordinary models: with no load subdirs the helper returns its input, and detection failures still degrade to root-only. * Let the model routes see the same cached snapshots training does Two ways /api/models disagreed with the training resolver, both reachable from resume: _model_config_inspection_target probed only the snapshot root, so a cached Spark-TTS/BiCodec copy answered "Selected cached model is no longer available" for a cache the training resolver accepts, and the exact-snapshot remote-code scan could fail with it. It now uses the shared with_load_subdirs helper. The model_snapshot_repo_id guard used the owner/repo-only regex, so resuming or scanning a namespace-less Hub model such as gpt2 or bert-base-uncased returned 400 before the snapshot could be inspected, even though hub.utils.paths .is_valid_repo_id and the picker both allow the one-segment form. That call site now uses the shared validator. The other five uses of the local regex predate this branch and are left alone. Reverting either fix fails one of the new tests and nothing else. * Get the resume refusal reason all the way to the user 300fe6321 fixed the diagnosis on POST /api/train/start, but the History UI never reaches it. can_resume: false hides the Resume button outright, and in the one window where the server could answer, resume-training-run.ts throws its own "Only stopped or errored runs with a saved checkpoint can be resumed" before sending any request, so no start call is ever made. For a run whose checkpoint is intact and whose pinned snapshot was evicted, that sentence is the wrong diagnosis, which is what 300fe6321 removed server-side. Carry the reason on TrainingRunSummary and prefer it in that guard. The field is optional and defaults to None, so old clients are unaffected, and it is only computed for rows already known unresumable. A checkpoint problem still reports None, leaving the client's existing wording for that case, and a failure inside the gate is swallowed so History still renders. Reverting either half fails one of the new tests: the summary stops carrying the reason, or the client stops preferring it. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Honour load subdirs on the resume pin and keep the selected model cache copy Two follow-ups to the cached-snapshot work. The resume branch of _reject_untrainable_model_request still probed the pinned snapshot with a hardcoded (config.json, adapter_config.json) tuple, so a repo that loads from a subdirectory (unsloth/Spark-TTS-0.5B keeps everything trainable under LLM/) resolved to None. Offline that became a 409 hf_model_not_cached_offline for a snapshot present on disk; online it fell through to the remote metadata round trip the pin exists to avoid. It now uses with_load_subdirs like the other cached-snapshot probes. The cache reconciliation effect took the first usable inventory row for the selected repo. A repo can be present under more than one HF cache root, so that silently retargeted an explicit selection at a different copy on the next inventory tick. The selection logic moved to a pure module and now prefers a row whose path matches the current selection, falling back to the first usable row when there is no selection or the selected copy has gone. Tests: test_resume_pin_load_subdirs.py (5) and model-cache-reference-selection.test.ts (8); reverting either fix fails them. * Revert the model cache reference preference; it cannot fire The frontend half of 2c2a95309 assumed the model inventory can return more than one usable row for a repo. It cannot. hub/services/models/cache_inventory.py::_scan_cached_models collects into seen_lower keyed by repo_id.lower() and resolves collisions with _prefer_cache_row, and _dedupe_local_models keys hf_cache rows by (model_id, model_format, format_variant). Both return one row per repo, and a live check with the same repo under four HF cache roots got exactly one row from /api/hub/cached-models and one from /api/hub/local. With a single usable row usable.find(pred) ?? usable[0] is identically usable[0], so the change was a no-op and its tests asserted an array shape the API cannot emit. The resume pin fix in the same commit stands: that one was reproduced against the real unsloth/Spark-TTS-0.5B cache and reverting its hunk restores the 409. * Report the resume refusal that actually happened 300fe6321 and a98e721f4 set out to stop a provenance refusal being reported as a checkpoint problem. They overshot. Both sites asked resource_provenance_resume_blocker whenever can_resume_run said no, but that function refuses for several reasons and the blocker is computed independently of which one fired. initialize_resource_provenance writes {version: 1, status: pending} at the start of every run, so the blocker answers "The model revision used by this run was not attested." for any Hub-model run, including one whose checkpoint is simply missing. That is the more common way to be unresumable, so the change traded one misdiagnosis for another and the new one covered the larger population. Both sites now gate on has_resume_state, the same discriminator can_resume_run short-circuits on: no saved trainer state means the checkpoint is the cause and the client's own wording is correct; an intact checkpoint means a refusal really is provenance's doing and the specific reason is worth surfacing. test_resume_reason_matches_cause.py pins it; reverting either guard fails it. The route contract assertion is over the AST rather than the source text, because a substring search is satisfied by the explanatory comment beside the code and passes with the guard deleted. Two assertions in test_resume_blocked_reason_surfaces.py encoded the old behaviour and are corrected, including the docstring claiming History survives a raising gate: can_resume_run calls the same gate unguarded one line earlier, so that only holds when it short-circuits first. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Scope the import-hoist __all__ skip to package __init__.py 68bce3934 skipped any name listed in __all__. The comment said "the whole point of a package __init__", but compare() never consulted the path, so the skip applied to every module defining __all__ -- 27 non-package modules exempting 224 names, unsloth/models/_utils.py alone 74. The cost is not just breadth. lint-ci.yml names rename-clash as one of the two bugs this tool exists to catch because ruff and pyflakes miss it, and the skip disabled that detection for any name in __all__. Adding a name there became a one-line, reviewer-invisible way to switch the check off. The self-test I cited did not cover this: --self-test reported ALL PASS with the skip hunk reverted, because every case ran through the "<name>" path placeholder and none defined __all__. Cases may now carry their own path, and three new ones pin the behaviour from both sides -- a re-export in a package __init__ is allowed, the same shape in an ordinary module is still blocked, and a new import absent from __all__ is still blocked in an __init__ too. Deleting the skip fails the first; widening it back fails the second. Removes two genuinely unused imports the correctly-scoped check then found in test_resume_blocked_reason_surfaces.py. * Make deleting a run's artifacts reversible until the row is gone DELETE /runs/{run_id}?delete_artifacts=true is new in this PR; on b41b819a4 the endpoint only calls delete_run and never touches the filesystem. As written it rmtree'd the output directory and then deleted the row, so a failing row delete left the artifacts destroyed and the row alive with output_dir still populated. storage/studio_db.py opens SQLite with Python's default 5s busy timeout and sets no busy_timeout pragma, so a writer holding the database longer than that is enough. The state that leaves is the real problem: a row whose artifacts are silently gone is indistinguishable from the legitimate keep-history outcome the same endpoint produces for a shared output directory, so the user cannot tell which happened. A retry does recover, but only because a missing directory is treated as success, which is incidental rather than designed. _delete_run_output_dir now performs a same-parent rename to .<name>.deleting-<uuid> and returns the staged path. The run is logically gone the moment that succeeds, but the bytes survive until delete_run commits; a failure restores the rename and the operation rolls back whole. The active and shared guards still run first, inside the same lifecycle guard, so neither can be raced. Three assertions in the author's test_training_history_delete.py move to the new return shape; their subjects, the lifecycle guard and the shared-output recheck, are unchanged and still pass. * Keep cached-snapshot resolution off the network 590ac9f22 routed the cached-snapshot resolvers through security_load_subdirs, which calls detect_audio_type. That function only skips its remote tokenizer fetch when local_files_only is set, and the new callers did not set it, so _resolve_model_snapshot and the two cache-pin sites -- all pure filesystem work before -- gained a hub round trip with no timeout in front of them. On a slow or hung hub, asking whether a snapshot is already on disk could stall. security_load_subdirs gains an opt-in local_files_only, default unchanged so the security scanner keeps the remote answer it wants. with_load_subdirs passes it: the subdir layout is a property of the snapshot on disk, so the local answer is also the correct one there. A side effect worth noting: a network failure previously raised straight past the YAML registry fallback, because security_load_subdirs wraps both branches in one try. Asked offline, detection reports nothing instead of raising, so the fallback now gets its turn. That comment-versus-code mismatch is byte-identical on b41b819a4 and is left alone, but it is pinned by a test so it stays a decision rather than a surprise. Three existing fixtures stubbed security_load_subdirs with a two-argument lambda and silently lost the expansion through the helper's except; they now carry the new keyword. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Pin the two load-subdir sites nothing was guarding An audit ran the full backend suite against ca7c72e75's three sites reverted one at a time. Reverting core/training/provenance.py or core/training/worker.py left 17204 passing tests green with a byte-identical failure set: both shipped undetectable. Only routes/training.py was caught, and only retroactively, by a file added in a later commit. Neither is decorative. For a subdirectory-loading repo the provenance site turns a snapshot present on disk into "The exact model snapshot for this run is no longer available." and refuses the resume; the worker site either errors with "The cached model snapshot selected during preflight is no longer available." under strict resume or, without it, silently drops the pin so the load goes back to the Hub. test_subdir_pins_are_guarded.py covers both, at the helper and at the user-visible gate. Reverting the provenance expansion fails 3 of the 6; reverting the worker expansion fails the non-strict pin test. Empty and wrong-subdir snapshots are still rejected, so the widening cannot mask an unusable cache. Also adds the debug line the shared except never had. Degrading to root-only is fail-closed everywhere, but a genuine cache permission or corruption fault reaches the user as "your cached model isn't cached" with no diagnostic, and four sites now share that handler. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Bind has_resume_state in the alternative-layout import fallback routes/training.py imports its helpers in a try and repeats the block under except ImportError for an alternative on-disk layout. The resume diagnosis fix added has_resume_state to the primary list only, so wherever the fallback runs the name is undefined and a resume request with intact checkpoint state and a failed provenance check raises NameError -- a 500 in place of the refusal reason that change existed to produce. test_route_import_fallbacks_agree.py compares the two blocks across every module under routes/, so the class is covered rather than this one instance: any name imported from the same module in the primary branch has to appear in the fallback too. Removing has_resume_state from the fallback alone fails it. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Let provenance read metadata off an MLX model _object_value tried the mapping protocol before attribute access. mlx.nn.Module subclasses dict (MRO: Module -> dict -> object), so every probe on a real MLX model took the dict branch and answered None, and both MLX paths added for attestation were unreachable: getattr(model, "_unsloth_quantized_source") -> 'runtime' _object_value(model, "_unsloth_quantized_source") -> None _loaded_model_is_4bit(model) -> False _loaded_model_refs(model) -> set() unsloth_zoo does record the metadata (mlx/loader.py sets _unsloth_quantized_source at 2748 and _hf_repo / _unsloth_base_commit_hash at 4047-4050); provenance just could not see it. Net effect on Apple Silicon: a runtime-quantized run -- the Studio default of a 16-bit repo with the 4-bit toggle on -- never attested, with or without a cache pin, so its checkpoints were never resumable. Read the attribute first and keep the mapping lookup as the fallback, which the plain dicts flowing through here (quantization_config, _unsloth_quantization_policy) still need. Verified with real MLX LoRA runs, provenance status before -> after: SmolLM-135M-4bit unpinned incomplete -> attested / prequantized_4bit Qwen3-0.6B 4bit pinned incomplete -> attested / runtime_4bit Qwen3-0.6B 4bit unpinned incomplete -> attested / runtime_4bit Qwen3-0.6B 16bit unpinned incomplete -> attested / unquantized resource_provenance_allows_resume returns True for all four afterwards, and _loaded_model_refs now resolves the repo/commit pair it previously missed. The mlx doubles in test_training_provenance.py are SimpleNamespace, which is not a dict subclass, so they exercised a branch no MLX model reaches. The new test parametrizes over a plain object, a dict subclass, and a real mlx.nn.Module; the dict-subclass case reproduces the bug without MLX installed, so it guards on CI too, and the MLX case importorskips off Apple Silicon. Prequantized snapshots that are not uniformly 4-bit (an 8-bit mlx-community conversion, mixed per-layer widths) still do not attest. That is a separate cause in _snapshot_declares_quantization and is left alone here. (cherry picked from commit 8af94dc1fd225905aa031d08b59ad752b20112f0) * Tighten comments across the train-page rework Collapse multi-line comment blocks to their essential reasoning, drop comments that restate self-explanatory code, and shorten section banners. Comments recording a non-obvious why (reproduced bugs, upstream quirks, deliberate tradeoffs) are kept, just stated in fewer lines. Comments and docstrings only: verified with an AST comparison against the previous tree, so no code changed. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Tighten comments across the remaining studio files Collapse multi-line explanations to a single line and drop a redundant section marker. Comments only, no code changes. * Fix offline dataset selection and training review regressions Resolve cached dataset configs and splits from trusted local metadata with a localized validated manual fallback. Require an explicit split for cached offline training while preserving remote and streaming format checks. Refresh the shared dataset inventory after full recipe completion so new artifacts appear on Train. Keep shared output deletion atomic and restore legacy dataset format column ordering. Keep dataset completion attestations on v2 while writing ordinary manifests in downgrade-compatible v1 and migrating prior ordinary v2 records at startup. Return stable codes for rejected training model selections. * Adapt the grad-norm payload test to this branch's config shape #7917 added tests/training-start-payload-grad-norm.test.ts against main's TrainingConfigState. This branch moved hfToken out of that state into a second parameter on buildTrainingStartPayload, dropped datasetUserTemplate and datasetAssistantTemplate, and added 11 required fields, so the merged file did not typecheck. Staging CI caught it; the local run had not covered the second merge yet. The literal now spreads initialTrainingConfigState and keeps only the fields the test actually varies, so a future field addition does not break it again. Still non-vacuous: reintroducing max_grad_norm: 0.0 in the mapper fails it with "max_grad_norm must be absent, not null and not 0", which is the regression the file exists to catch. typecheck clean, 634 frontend tests pass. * Scope pending training cancellation to its start request Keep the request identity until cancellation or exact status reconciliation completes. Reject cancelled starts before worker spawn and stop or reset only the job owned by that request. Fence the process-start race, clean up adoption failures, and keep unrelated jobs untouched. Remove explanatory comments added with the earlier dataset and manifest fixes. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update the training-start contracts to the scoped cancellation shape Four assertions in tests/studio/test_hf_token_validation_tick_contract.py still described the pre-scoping implementation, so the Studio contract suite was red at 8443285df. isTrainingStartPending gained a startRequestId disjunct. Asserted per term rather than as one exact string, so the expression's formatting is not the contract. stopTrainingRun and dismissTrainingRun replaced the expectedJobId locals with trainingStopScope / runtimeMatchesStopScope, and the request now sits inside a try, so the transport slice cut at the scope === null early return instead of the call. Re-sliced on the try/catch and renamed the reads. The blanket "setStopRequested(false) not in stop" no longer holds: the start branch clears the latch on purpose so a pending-start cancel stays retryable. Split per branch instead, which is a stronger contract than before -- the job branch must still keep the latch, and the start branch must still clear it. Non-vacuous, checked both ways: dropping the startRequestId disjunct fails the pending test, and adding setStopRequested(false) to the job failure branch fails the cancel-lease test. 154 tests across the four contract files pass; tests/studio 2561 passed. * Offer the cached dataset options the start request will accept local_options.py starts the subset and split options the picker shows, but it was written with a different grammar from TrainingStartRequest, so it disagreed in both directions. _SPLIT_RE was \w+(?:\.\w+)* and dropped the hyphen, so a real split name such as train-clean (LibriSpeech) never reached the picker even though the start request accepts it. An offline user had to type it by hand. \w is Unicode-aware in Python, so trein with an accent was offered and then rejected by the ASCII-only split validator. _CONFIG_RE excluded only filesystem-hostile characters, so a config name containing a space was offered and then rejected by the subset validator. Both turn a click on an offered option into a 422. Both patterns now mirror _check_subset and _check_split_name, which are the authority, and _valid_option rejects ".." anywhere rather than only as a whole segment, matching the split validator. New test drives the real pydantic validators rather than restating their regexes, and asserts over the normalized value, since _valid_option strips and it is the stripped string that gets offered. Reverting either pattern fails 6 of its 14. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Report a failed artifact purge instead of claiming the delete succeeded Deleting a run with its artifacts stages the output directory with a same-parent rename and purges it once the database row is gone. The purge swallowed OSError while the response still said artifacts_deleted=true, so a failed rmtree stranded every byte under the hidden randomized .{name}.deleting-<hex> name: the row is gone by then, so nothing points at it and no retry can rediscover it. The purge now reports whether the bytes are actually gone, and puts the directory back under its own name when they are not. The response says the artifacts were kept, with a purge_failed reason, and the history grid's existing artifacts_deleted check surfaces it. Same window, other half: when the row delete and the restoring rename both fail, the response now names the staged path instead of raising a bare 500 that leaves the artifacts unreachable. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Keep offering cached subsets whose only oddity is a dotted name Aligning the picker's grammar with TrainingStartRequest went one step too far: _valid_option started refusing ".." anywhere, for both fields. _check_split_name does reject ".." outright, so that is right for splits, but _check_subset only constrains the charset, which means a config directory named v1..v2 is perfectly startable and the picker was hiding it. That is the same bug as offering an option the start request rejects, just pointing the other way. The ".." refusal is now scoped to split names. Neither charset admits a separator, so the bare "." and ".." names remain the only traversal shapes left to refuse for either field. test_processed_cache_options_are_local_deduplicated_and_non_train_capable pinned the pre-change behaviour for both names in its fixture. Its "config with spaces" entries stay dropped, since _check_subset rejects the space and offering it turns a click into a 422, and "v1..v2" comes back. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com> Co-authored-by: shimmyshimmer <michaelhan2050@gmail.com> Co-authored-by: Daniel Han <danielhanchen@gmail.com> Co-authored-by: Unsloth <michaelhan@Michaels-MacBook-Pro.local> Co-authored-by: danielhanchen <unslothai@gmail.com> Co-authored-by: Bardia Koopah <bkoop2003@gmail.com>	2026-08-06 04:26:05 -07:00
Michael Han	6b0035b8e6	studio: apply saved per-model settings on API loads, add API monitor (#7473) ... * studio: apply saved per-model settings on API loads, add API monitor Per-model settings were mirrored to the server as two fields only, llama_extra_args and max_seq_length. Everything else lived in browser localStorage, so a model loaded by an API request came up with app defaults for context length, KV cache dtype, speculative decoding, tensor parallel and GPU placement. Store the full config server side and map it onto the same LoadRequest the picker builds, so a remote load and a picker load of the same model produce the same command line. Entries are keyed per quant, falling back to the bare repo id so existing entries keep resolving. Also moves the API monitor out of the settings tab: a full page at /api-monitor, plus a floating panel that opens itself when traffic arrives, and a settings page per model reachable from the Hub. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * studio: harden per-model settings against review findings and fuzzing Review findings (PR #7473): - Look up overrides under the concrete load path with its quant, not just the advertised repo id, so local folders and non-active HF caches are found. - Carry bare-repo launch flags into the first per-quant save. Auto-switch prefers the qualified entry, so without this the flags were silently dropped and no UI could show or restore them. The bare id is only derived when the suffix looks like a quant, so a Windows drive letter is not split. - Drop a saved gpu_ids pin that no longer resolves instead of 400ing the whole load. A pin outlives the machine it was made on. - Build the displayed API base from getApiBase() on desktop; the Tauri webview origin is not the API server. - Keep a partial download's isDownloaded when opening settings, so the loader still reports download progress. - Only prefer the loaded quant when the loaded model is this row. Q4_K_M exists in most repos, so an unguarded match targeted the wrong variant. Found by simulation: - A lone surrogate in a chat template raised UnicodeEncodeError on the byte check, an unhandled 500. Now a validation error, in all three call sites. - _bounded_int accepted bools as GPU ids, truncated fractional floats, and raised OverflowError on Infinity, which json.loads accepts. - api_monitor stored a non-string model verbatim; the monitor page then threw on toLowerCase and rendered nothing. Coerced at the boundary and the filter no longer trusts network data. - The overlay store now uses storage that cannot throw. Safari private mode and blocked-cookie origins make localStorage throw on access, which broke the opt-out toggle. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * studio: address second review round on per-model settings - Probe actual Vulkan devices in the GPU reconciliation helper. Vulkan ordinals are their own index space, so resolve_requested_gpu_ids only rejects malformed ones; the helper said "usable" and the load then 400d on the ggml probe it had skipped. - Send explicit save/remove intent. A save whose config is entirely default carries no fields, which is shape-identical to "forget this model", so it was wiping launch flags the UI cannot show or restore. A bare model_id without the flag still removes, keeping the original contract. - Backfill existing per-model settings into the server map once after upgrade. Without it, settings saved before this change never reached the server, so an API load used app defaults while the UI still showed the model as remembered. Never overwrites a server entry and retries until it fully succeeds. - Stop polling the monitor endpoint when auto-open is off and the panel is closed. It cannot open or display anything in that state, so every open Studio window was polling every five seconds for nothing. * studio: address third review round on per-model settings - Fall back to a case-insensitive override lookup. The browser normalizes ids to lowercase before storing, so the backfill wrote "org/model:q4_k_m" while the resolver asked for "Org/Model:Q4_K_M" and never matched, which made the whole migration a no-op. Exact match still wins and an ambiguous fallback matches nothing, so two POSIX paths differing only in case stay distinct. - Record a detail revision only when a fetch actually started. requestDetail's in-flight guard can refuse, and recording anyway meant a revision that landed during an earlier fetch was skipped for good once updated_at stopped moving, leaving a terminal request showing a stale running payload. - Discard superseded settings opens. The GGUF variant lookup is async, so opening a second row while the first was pending let whichever finished last win, and the page could then save or load settings for the wrong model. - Raise the model_id cap to PATH_MAX plus a quant suffix. A local model's id is its filesystem path and LoadRequest.model_path is unbounded, so the old 512 limit 422d the server sync while the local save succeeded, leaving the UI showing settings the API would never apply. * studio: only open the API monitor for real API clients CI caught this: the Chat UI Playwright run failed because the floating panel opened during an ordinary chat turn and its Expand button covered the composer's Send button. The panel opened because Studio's own chat goes through the same tracked endpoints as the OpenAI-compatible API, so any conversation looked like API traffic. That is wrong regardless of the click interception: this panel exists for when Unsloth is being used as an API server, not when someone is using Unsloth. Record on each monitor entry whether the caller authenticated with an sk-unsloth key rather than a UI session, and auto-open only for those. The discriminator already existed for other routes; this reuses it. * studio: address fourth review round on per-model settings - Never case-fold a filesystem path when looking up an override. Two POSIX paths differing only in case are two different files, so a near miss must load defaults rather than replay another model's context and GPU pin. Repo ids still fold, which is what the migration needs. - Match a quant suffix against the loader's own quant pattern instead of a length heuristic. "/models/foo:bar.gguf" is one valid POSIX filename, and splitting it grafted /models/foo's launch flags onto an unrelated model. - Retry a load once without the saved gpu_ids when the loader rejects the pin. The pre-flight check cannot mirror every rule the loader applies (a Vulkan diffusion GGUF refuses GPU selection outright, and the rules move), so this stops chasing them one at a time: a stale placement preference must never be the reason a request cannot be served. - Make an explicit remove win over config fields sent in the same payload. - Seed only finished requests on the monitor's first snapshot. A request still running when Studio loads is traffic the user has not seen, not history. - Nudge the detail effect when the in-flight guard refuses a fetch. Nothing else changes its deps when the older fetch settles, so a terminal reply could stay truncated forever. - Invalidate pending row lookups when opening settings from a detail card, not just from a row. - Mark the covered detail pane inert so it leaves the focus order. - Refuse to open settings for a variant-required GGUF whose quant could not be resolved: the picker matches variants exactly and would never find the saved config, while the API falls back to the bare key and would apply it. - Mirror to the server only for GGUFs. The auto-switch resolver indexes GGUFs, so a safetensors config was being advertised as applied on API load when no API request could ever apply it. * studio: clear server overrides for models evicted from local storage Saving a config can push the browser's map over its entry or byte budget, at which point older models are silently dropped and the save still reports success. Their server-side overrides survived, so API loads kept applying settings that nothing in the UI showed any more and nothing could forget. savePerModelConfig now reports what it evicted, and the caller clears those server entries alongside the one it saved. Removal also had to resolve the way lookup does. Storage keys are normalized, so an evicted entry comes back lowercased while the server may hold the repo's real casing; deleting only the literal key would leave the entry a load still resolves to. Read and remove now share resolve_model_override_key, so what a load applies and what forgetting clears cannot disagree. Path ids still match exactly, so one file is never forgotten by clearing its case-variant neighbour. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * studio: pin the GGUF mirror gate and eviction cleanup with contract tests Both were flagged in review with no test holding them in place. Also corrects the gpu_ids preflight docstring: it claims to mirror every rule the loader applies, which it deliberately does not, and the retry below is what covers the rest. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Resolve the override key on save and match standalone gguf settings The remove branch already resolved the key a load would use, but the save branch wrote payload.model_id literally. The browser normalizes casing before storing, so a backfilled key and a later UI save left two entries for one model; with two equivalent keys present resolve_model_override_key finds no unique match, so any third casing resolved to no override at all and the model silently loaded with defaults. A standalone .gguf gets variants=() from the resolver, so variant is None and only the bare ids were tried. The picker keys the same file by the quant label it derives from the filename, which is never empty, so those settings lived under <path>:LABEL and nothing reached them. Try the filename-derived key after the variant-qualified ones and before the bare ones, so older bare entries still work. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Match server overrides by identity during the one-time backfill app_settings carries no schema version, so an install that predates identity normalization holds rows keyed by whatever id was typed, such as Unsloth/Repo-GGUF:Q4_K_M, while this browser only ever stores the folded form. The exact property lookup therefore reported "not on the server" for a row that is, and the backfill PUT over it, replacing server settings the file documents as the newer authority. The migration runs once on every existing profile, so this lands on exactly the upgrades it was written to protect. Fold both sides before comparing, splitting on the last colon because a quant label never contains one. A repo id and a Windows path fold, a POSIX path does not, which is the same rule the backend resolves by. Verified with the real module under node: the legacy-casing, variant-casing and Windows-path cases go from overwriting to skipping, a second run stays clean, and a genuinely new model, a different quant of the same repo, a POSIX path differing only in case, and a bare legacy key all still migrate. * Fold case-insensitive path ids the way the browser already does resolve_model_override_key refused the case fallback for every filesystem path, but only a POSIX path is case-sensitive. A Windows drive path, a UNC share and a WSL drive path each name one file whatever the casing, and the browser folds exactly those three before storing. A Windows user's migrated entry was therefore keyed lowercase while an API auto-switch resolved the same file with its on-disk casing, so the lookup missed and the saved launch flags silently stopped applying until the settings were saved again. Fold those three shapes here too, normalizing the separator as the browser does so C:/Models/Foo.gguf and c:\models\foo.gguf agree. POSIX stays case-sensitive, /mnt/data stays an ordinary mount rather than a WSL drive, and an ambiguous fold still matches nothing so a load takes defaults instead of guessing. The existing Windows test asserted the opposite. It carried no rationale, unlike its POSIX sibling, and get_model_override's docstring already scopes the rule to POSIX, so it read as an over-generalisation of the POSIX case. * Close clearApiMonitor The merge that brought main into this branch dropped the closing brace, so the function body ran straight into the interface declared below it and the frontend did not compile at all: tsc reports TS1005 at the end of the file and vite fails the build. Reproduced against the pushed head and clean with the brace restored. * Carry legacy flags across a scanner-derived GGUF variant label A .gguf whose filename holds no recognizable quant token still gets a label from the scanner, which falls back to the filename stem, so the UI stores keys like "/models/custom.gguf:custom". _bare_model_id accepted only known quant tokens, so the first per-quant save did not carry over llama_extra_args stored under the bare id, and auto-switch prefers the qualified entry, so those flags were silently dropped with no UI able to restore them. Accept a suffix that is exactly the label the scanner derives for that filename. Requiring the head to be a .gguf and the suffix to match exactly is what keeps an arbitrary colon-containing POSIX path out: "/models/foo:bar.gguf" splits to a head that is not a .gguf. The filename is taken by splitting on both separators, since a "C:\\..." key is written on Windows but may be read back by a backend that is not. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Keep model lifecycle rows out of the request statistics A load, unload or download is recorded in the monitor but is not an HTTP call. It reads as running for as long as the load takes, so it was counted as an in-flight request with no client waiting, and a multi-minute download was folded into Avg latency and the error rate. The backend already excludes these rows from active_count for the same reason, so the page was also disagreeing with the number the API itself reports. Requests counted them too, so that is now the non-lifecycle count rather than the raw entry count. Also limit the API-reach sentence on the Hub settings page to GGUF models. The Hub opens that page for every downloaded model, but ModelConfigPage mirrors settings to the server only when target.isGguf, because auto-switch indexes GGUFs only, so a safetensors user was told the settings apply to an API request that cannot reach them. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Accept bpw variants, fold POSIX quant suffixes, migrate bare GGUF configs Three gaps the previous round left, all in the same key-resolution rule, so the rule now lives in one place as split_quant_suffix. A quant label may carry a bits-per-weight modifier, because two files at the same base quant are kept distinct by it: utils/models/model_config.py preserves IQ4_XS-3.53bpw while hub/utils/gguf.py strips it, and both forms reach the override keys. The known-quant pattern accepted neither, so _bare_model_id missed the bare entry and the first qualified save dropped its launch flags. On POSIX the browser lowercases the quant but keeps the path casing, so a migrated "/models/Foo:q4_k_m" was unreachable from the scanner's "/models/Foo:Q4_K_M". Only the quant suffix folds now, and only when it really is a quant, so "/models/foo:Bar.gguf" stays a distinct filename and the path itself stays case-sensitive. A standalone .gguf picked directly has no quant to choose between and is stored with a null variant. The backfill filter read that as safetensors and skipped it, and the done flag is set on the same pass, so those settings stayed browser-only for good while auto-switch kept loading the model with defaults. * Let remove win over flag validation, and make Clear log clear shared rows An explicit remove ran the launch-flag validation first, so a form still carrying a rejected flag raised a 400 and left the override in place. Nothing is stored on that path, so there is nothing to validate; remove now short-circuits it, which is what the branch below already claims to do. Clear log dropped only the caller's own rows, but a lifecycle row is shared: it is visible to everyone and owned by no one, so those rows survived and the reload straight after the click brought them back, leaving the button visibly ineffective. Deleting them is not an option either, since that erases another caller's history. They are now hidden per subject, so the clear is true for that caller and harmless to the rest. A shared row that is still running is live state rather than history, so it stays visible, and the hidden ids are pruned against the ring buffer so they cannot accumulate. * Move the lifecycle labels out of the lazily loaded monitor page The overlay is mounted from __root.tsx and imported two label helpers from the page, so the page and its dependency graph were pulled into the eagerly loaded bundle and the route's lazyRouteComponent bought nothing: every route paid for the monitor page even when it was never opened. Measured on a production vite build, the async api-monitor chunk was 0.20 kB, meaning the implementation had landed in the main bundle. The helpers now live in their own module. The same build gives an 18.83 kB api-monitor chunk and a main bundle 18 kB smaller (3.9 kB gzipped). * Order override writes per model Saving twice quickly, or saving while the one-time backfill is still running, started independent requests with no sequencing, so the older response could commit last and resurrect the entry the newer one meant to replace or remove. An API-driven load then applies context or GPU settings the user has already changed, with nothing in the UI showing it. Writes now chain per override key. The chain hangs off the settled tail, so a failed write cannot cancel the next one, and only the last writer clears the slot so a queue that is still building keeps its order. Different models still overlap. Verified against the real module under node: two saves for one model with the first made slow commit oldest-first and never overlap, where the previous version committed them in the wrong order; a rejected write still lets the next succeed; and two models still run concurrently. * Key write queues by identity, and reach unknown GGUF labels in either casing Two holes in the previous two commits, both found by the same review round. The per-model write queue keyed on the literal spelling, so the backfill's legacy casing and a UI save's normalized one opened two queues for one model and raced exactly as before. It now keys on the folded identity, which is what the backend resolves by. A .gguf with no recognizable quant token is labelled by its filename stem, and v2 storage lowercases that label while the scanner probes with the filename's own casing. Folding only recognized quant labels therefore left the migrated entry unreachable for precisely the files that need the stem fallback. The suffix rule now also accepts a case-insensitive match against the label the scanner derives for that filename, which keeps an ordinary colon out because the head still has to be a .gguf. _bare_model_id drops onto the same shared rule rather than repeating half of it. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Guard the backfill, the detail settings entry point and the detail retry Three separate reports, all confirmed against head. listPerModelConfigs reported future-schema records. loadPerModelConfig refuses to apply one and eviction refuses to drop one, because this client cannot interpret that schema, so handing it to the backfill would persist a partial reading of it server-side and let an API-triggered load apply settings the same client will not apply locally. It is skipped there now, matching the other two paths. The detail view's on-device card passes a null variant while its own lookup is pending or after it failed, and this entry point opened the editor anyway. That saves a bare-model config, which the picker never finds because it matches variants exactly, while the API's bare-key fallback would apply it. openModelSettings already refuses with a toast for exactly that reason; this path now refuses the same way. A failed detail fetch was never retried. The revision is recorded when the fetch starts, and on failure the entry stays missing, so selectedIsMissing does not change and a terminal row's updated_at does not advance: nothing was left to re-run the effect, and the full prompt and reply stayed unavailable until another row was selected. The in-flight flag settling is the trigger now, and the attempt count bounds it, because the usual failure is an entry that has aged out of the ring buffer and will never arrive however often it is asked for. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Point the lifecycle contracts at the module the labels moved to Extracting the labels out of the monitor page left two contracts asserting they were still in it, so the staged run went red on all three platforms. They read the new module now, and the page contract additionally pins that it imports from there rather than redefining them. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Re-read local state before backfilling, and stop advertising Ollama as API loadable The backfill wrote the snapshot it took before fetchModelOverrides resolved, so a save or a forget during that round trip was undone: the write is queued behind the interactive one and commits last, leaving the browser showing the new settings while an API load applied the old ones. Each write now re-reads the model's current local config and skips it if it has gone or gone back to defaults. Verified against the real module under node: the write carries maxSeqLength 9999 where it previously carried the stale 1000. target.isGguf was also standing in for "an API request can load this". It cannot for an Ollama model: local_model_resolver skips Ollama's scanner on purpose, so those models are never in the auto-switch index, yet the mirror ran and the settings page told the user the API would apply them. The target now carries apiLoadable, set from the inventory source the row already has, and both the mirror and that sentence read it. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Key the Hub's per-model settings by the repo id, not by the load path A repo cached outside the active HF cache reports load_id as its snapshot path (cache_inventory), which is what the loader needs, but the chat picker and the auto-switch index both name that repo by repo_id. The new Hub settings page was saving under the load id, so the settings landed on a key no other load reads: the picker, an auto-load and an OpenAI-compatible request all fell back to defaults, and a server override already stored under the repo id could win against the save. ModelPickTarget now carries configId for the case where the storage identity is not the loadable one. Every read, write and server mirror in ModelConfigPage uses it; the chat template and GGUF header probes keep target.id, since they have to open the model. The Hub sets it for cache rows and resolves its own load through the same helper, so a config saved from the settings page is the one a later load finds. Rows whose load id is already their identity, which is every local row and every repo in the active cache, are unaffected. Verified against the real per-model-config module under node: saved under the snapshot path, a picker read reports remembered=false and the default max sequence length; saved under the repo id it reports remembered=true and 8192. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Tighten the comments across the API monitor and per-model settings work Pass over every comment this branch touches. Collapse the multi-paragraph rationales to the point they were making, drop prop docs that only restated the prop name, and reflow the rest onto fewer lines. No code changes. * Recognise an absolute path id in either platform spelling _is_abs_path_id decides whether an id is a host path that must not be published through /v1/models, and it asked pathlib.Path, which follows the running OS. A Windows backend therefore read "/home/me/x.gguf" as a relative name and a POSIX one read "C:\models\x.gguf" the same way, and in both cases the path was advertised verbatim as a model id. Ids outlive the machine that wrote them: settings sync, a WSL session and a copied config all carry the other platform's spelling, which is why the model-override identity in this PR already folds Windows drive, UNC and WSL paths. The backend now agrees with it, reading the value as both a POSIX and a Windows path. Neither reading can misfire on a repo id, which has no leading separator, drive letter or UNC prefix. The two tests this fixes on Windows are older than this PR; the new one pins the contract in both directions. * Give the idle-unload test a wall-clock deadline The drive loop waited a fixed 200 iterations of a 10 ms sleep. Windows rounds that sleep up to the roughly 15.6 ms scheduler tick, on this loop and on the idle loop under test alike, so the unload got far less real time there than the count suggests and the test failed on the Windows runner for being slow rather than wrong. It now waits on time.monotonic with a generous ceiling and still breaks as soon as the KV file is gone, so the fast path costs nothing. The test is older than this PR; the deadline is the only change to it. * Split a quant suffix the way the backend does, and gate the detail card too The backfill took a key's identity by splitting on its last colon, so anything else that ends in one was read as a quant separator. A Windows path made "C:\models\foo.gguf" into model "C" with variant "\models\foo.gguf", and an ordinary colon inside a POSIX filename folded "/models/foo:Bar.gguf" and "/models/foo:bar.gguf" onto one key, so whichever of the two was already on the server made the other look migrated and left its API loads on defaults. splitQuantSuffix now mirrors split_quant_suffix on the backend: the suffix has to be a known quant label, with or without a bits-per-weight modifier, or the head has to be a .gguf carrying a stem label. Checked against the backend over twelve keys, including every case above, with identical answers on both sides. Settings also opens from the on-device detail card, and that constructor never set apiLoadable, so an Ollama model reached the server mirror and the "API loads use these settings" line from that entry point even though the auto-switch resolver skips Ollama's scanner. It now reads the same source the row menu does. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Judge the config storage actually keeps, not the one on screen savePerModelConfig normalizes before deciding whether a config is default, and the runtime hands the settings page Speculative Decoding "auto", which canonicalizes to null. The page judged the raw object instead, so a model sitting at defaults looked non-default: turning on "Remember for this model" reported saved while the local write had dropped the entry, reopening showed it as not remembered, and the mirror sent the server a speculative_type "auto" override the browser did not have. That disagreement between the two is the one thing the mirror is written to avoid. The page now normalizes once and uses that object for the default check, the local write and the server mirror alike. Driving the real module under node: the raw object reads as non-default, storage stores nothing, and only the normalized reading agrees with storage. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Shorten the comments on the newest round of changes Comment-only pass over the quant-suffix split, the storage-shape normalization and the wall-clock deadline in the idle-unload test. Same points, fewer lines. * Key one config per model, and stop the backfill replacing newer server state Five follow-ups on the per-model settings map. The one-time localStorage backfill read the override map once and then wrote each model in turn, so a save by another tab during that pass was replaced by this browser's older copy, against the migration's own "never overwrites" contract. Re-fetching per model would cost a round trip each; instead the PUT takes only_if_absent and the server tests and writes under one transaction. gpu_ids arrived unbounded and normalize_model_override de-duplicated it by scanning the list it was building, so a large authenticated array cost roughly 20x what the same work costs with a set (4.5s against 0.27s for a million entries). The payload now bounds the field to the number of ids the normalizer can store, and the dedupe uses a set. A settings target opened from the Chat model picker carried no apiLoadable, so the isGguf fallback mirrored an Ollama GGUF to the server. Ollama's blobs reach that picker as custom-folder GGUFs under a .studio_links / ollama_links dir, which local_model_resolver refuses to index, so the mirror advertised a load the API can never make. The picker, the sidebar editor and the backfill now all use the same classification. The Hub settings page compared the loaded model to settingsTarget.id, but a GGUF loaded from an inactive HF cache or straight off disk loads by path while /status reports the clean public id, so the page ignored the live launch config and showed saved or default values. It now also matches the settings identity and the public id the backend would report. A standalone .gguf gets a filename-derived format_variant from the inventory, so the Hub row menu stored its settings under <path>:Q4_K_M while the Chat picker, the detail card and the backfill all used the bare path. The row menu now uses the bare path too. Tests: publicModelId / residentModelIdMatches / isOllamaLinkPath / settingsGgufVariantForRow in studio/frontend/tests, the create-only write and the gpu_ids bound in studio/backend/tests, and the wiring in tests/studio/test_model_picker_contracts.py. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Read the concrete model key first, and fill settings in field by field Three follow-ups on the last round. The auto-switch load tried the advertised id before the concrete load path, so an entry under the alias shadowed the settings the user had just saved and kept shadowing them. The settings page keys every local row by the path being loaded, while the alias is derived: the /v1/models name a hand-written overrides PUT is written against, and for a loose .gguf only its filename stem. Each pair now reads the path first, and the alias, the bare ids and the older filename-label key are all still read after it, so a cached repo (keyed by its repo id, which is the alias) resolves exactly as before. publicModelId collapses two paths that share a filename or a directory basename onto one id, so the resident check added last round could mark the wrong catalog row as loaded and seed its editor with another model's live launch config, then save that under this model's key. The Hub page now records the loadable identifier /status reports, as every other status reader already does, so the literal comparison names one row; the public-id pass only accepts a namespaced repo id, the one collapse that cannot name two models. The override map shipped before this browser mirror did, holding only llama_extra_args and max_seq_length, so an upgraded install can have a server entry for a model whose context, KV cache, speculative and GPU settings live only in localStorage. The backfill read key presence as done, skipped exactly those models and then marked itself complete, so their API loads lost the settings for good. Filling them in from the browser would reopen the race the conditional write just closed, so the merge is the server's: the PUT flag is now fill_absent_fields, and studio_db merges field by field under the write's own transaction, where a stored value always wins. A fill with an entry already there also stops replaying that entry's stored flags through validation, so one denylisted since it was saved cannot 400 the one-time migration. Tests: the key order and the fill in studio/backend/tests, the collapse in studio/frontend/tests, and the wiring in tests/studio/test_model_picker_contracts.py. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Attribute API-key traffic on the lifecycle rows it creates The overlay opens on API-key traffic only, and record_lifecycle never set that flag. Auto-switch and auto-download run before the endpoint opens its request row, so a switch or a download that is refused never reaches api_monitor.start and its lifecycle row is the whole trace of the request. The monitor therefore stayed shut on exactly the failures automatic observability is for. The row now carries the attribution: a load takes it from the request that drove it, and auto-download passes it directly, since only an API request reaches that path at all. A manual unload and an idle unload are not API traffic and stay unattributed, so neither pops the overlay. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Treat a loaded standalone GGUF as resident despite its derived quant A loose .gguf keys its settings by the bare path with no variant, since the path already names the one file. The loader still derives a label from the filename and /status reports it, so the settings page compared a derived quant against a deliberate null, never matched, and withheld the live launch config from the very file that was loaded. Applying from that page could then write the saved values over what the resident model is running with. The variant equality is skipped for that case only. Every other target, a repo row or a directory, still has to agree on the quant, because there the variant is what tells two loaded copies apart. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Show the Hub the settings the model is really running with Two ways the Hub's per-model settings page could offer a resident model's saved or default values as if they were live, and then write them back over the running config on Apply. ModelConfigPage seeds its editable state from loadedConfig in a useState initializer, so it reads that prop once per mounted instance. The sidebar entry keys its instance on a signature of the live config; the Hub's keyed on the model and quant only. Open the page before /api/inference/status has hydrated, or while that same target is still loading, and loadedConfig flips from null to the live config after mount with nothing to remount on: the editor keeps the values it seeded from and Apply reloads the model with them. Both hosts now mount under one shared key that includes the live config, so the arrival of that config re-seeds the editor and a repeated poll of the same values does not. The live config itself comes out of the chat runtime store, and landing straight on /hub is the one entry point where nothing has applied the status yet: useChatModelRuntime has no mount sync and the chat page is a different route. The Hub's own status effect pinned the checkpoint and stopped there, so the resident check passed while kv cache, speculative decoding, tensor parallel and every GPU placement field still held their defaults. It now applies the whole status, the same call the chat runtime's refresh makes, and holds off when a load owns the store or an external provider is selected so it cannot fight either. * Pop the monitor open only for the traffic a caller made Lifecycle rows are shared so a load or a download shows up in everyone's monitor list, which is deliberate. Since they started carrying via_api_key they also carry the flag the floating panel auto-opens on, and that reached every authenticated subject: another logged-in browser sprang open for API traffic it had nothing to do with. The row now records the caller that drove it and reports the attribution only to them. Visibility is untouched, so the row still appears for everybody, and a subject-scoped Clear hides a shared row it owns rather than deleting it out of everyone else's history. Auto-download had the flag hardcoded on, reasoning that only an API request gets that far. Only a /v1 request does, which is not the same thing: Studio's own chat calls those same endpoints with a session JWT, so a chat that named a model this server does not have popped the panel open mid-chat, which is exactly what via_api_key exists to prevent. The attribution now comes from the request that asked for the download. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Require the scanner's own label before folding a .gguf colon suffix The frontend splitter accepted any suffix after a .gguf head, while the backend requires that suffix to be the label the scanner derives from the filename. A colon is legal in a POSIX filename and POSIX is case sensitive, so "/models/llama.gguf:Bar.gguf" and "/models/llama.gguf:bar.gguf" are two real, distinct files. The one-time backfill folded both onto one override key, since the variant half is stored lowercased, and then re-read the local configs by that key, so the first entry was sent twice, the second file's context and KV cache settings never left the browser, and the done flag was set anyway. splitQuantSuffix now ports extract_quant_label for a bare filename, shard suffix and float-precision fallback included, and takes the suffix only when it equals that label. Checked against the backend over twenty-nine keys with identical answers on both sides, up from twelve, and the backfill now migrates both files with their own settings. The identity helpers come straight from features/hub/lib/model-identity rather than the hub barrel, which also re-exports the download manager and its React components. Same bindings, and it puts the module within reach of a test. The 27 new frontend assertions cover the split case by case against the backend's answers, and pin the storage rule the backfill's re-read depends on: two spellings of one repo id or one Windows path keep a single record, a POSIX path keeps two, and importing the legacy load settings never adds a duplicate. * Point the parallel-slots contract at the shared config signature main added nParallel to sidebar-model-config.tsx's own configSignature while this branch moved that helper into config-signature.ts so the hub, the sidebar and the model config page key one editor instance the same way. The behaviour is unchanged, so the assertion follows the expression to where it now lives and pins the sidebar to the shared key. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Refresh Hub inference status after API-driven model switches The Hub read /api/inference/status once, on mount. An OpenAI-compatible request can auto-switch the resident model at any time, and nothing else on /hub reads status, so every 'loaded' marker and, more importantly, the per-model settings page kept describing the model that was resident when the Hub opened. The cost is concrete: with a stale checkpoint the newly loaded model fails settingsTargetIsResident, its settings page is handed loadedConfig=null, ModelConfigPage seeds the editor from saved or default values, and Apply reloads the model with them over the launch settings the API selected. Re-read on the moments this tab could have missed a switch -- regaining focus or visibility, and opening a model's settings page -- rather than polling on a timer. adoptResidentModelStatus already stands down for an external selection and for a load this tab started, so re-running it never fights the model the user is switching to, and applying an unchanged status leaves the live config identical, so the settings editor is not remounted under a draft. * Mirror per-model parallel slots to API loads and keep launch flags on eviction Parallel decode slots are a per-model setting the picker sends on every GGUF load, but the server-side override the OpenAI-compatible auto-switch path reads never carried them, so an API load of the same model fell back to the server-wide --parallel default. llama_extra_args could not stand in either, since --parallel is on the managed denylist. A config whose only change was the slot count also serialized to an empty payload, so the one-time backfill sent nothing and still marked itself done. Carry n_parallel through the serializer, the override schema, the normalizer and the load kwargs, and list it on the monitor so such an entry no longer reads as app defaults. Evicting a model to stay inside the browser's storage budget also sent the same full remove an explicit Forget does, which wiped llama_extra_args set through the settings API that no UI can show or restore, for a model the user never touched in that save. Eviction now clears only the mirrored fields; the route already carries stored launch flags over and drops the row once nothing is left. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Keep native-leased GGUFs off the API mirror and date the monitor's first snapshot Two identities the OpenAI-compatible auto-switch can never reach were being treated as if it could. A dropped or file-picked GGUF loads through a signed native-path lease, and /api/inference/status reports model_identifier as null for it, so the checkpoint the browser keys settings by is the bare file name the backend echoes back. local_model_resolver._build_index keys a standalone GGUF by its on-disk path and by its .gguf-stripped stem, so that name is never an index key: mirroring it wrote a server override no load can read, and the monitor's "Settings applied on API load" list then advertised it as live. The save gates on the lease token rather than the name, since the label falls back to a plain string with no suffix, and the one-time backfill, which has no token to read, goes by the identity shape. The floating monitor also wrote off every finished row of its first snapshot as history. That snapshot is not taken at mount: a hidden tab issues no fetch at all and an unreachable backend fails one, so the first API call of the session can start and finish before it lands and never open the panel. Date the backlog from when the poll stood up instead, on the server's own clock minus a browser duration, so a browser that disagrees with the server cancels out rather than replaying the whole ring buffer. A backend with no clock field keeps the old behaviour. The decision moves into its own module so it can be driven without a browser, which the overlay's .tsx dependency graph rules out. * Record a monitor row for a /v1 call refused by auto-download * Key a standalone GGUF's settings the same way on every surface llama_cpp falls back to _extract_quant_label(gguf_path) when a load names no variant, and /status echoes that as gguf_variant, so the sidebar wrote settings under <path>:Q4_K_M while the Hub row, the picker and the backfill used the bare path. The auto-switch lookup reads target_id before target_id:file_variant, so the bare entry always won and a sidebar edit could never reach an API load. The sidebar now nulls the variant for the settings identity, the same rule settingsGgufVariantForRow applies to a Hub row, sharing one definition. The displayed label still carries the quant. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Judge residency on settled state, not a snapshot Three places read the resident model from state that had moved on. The Hub's variant lookup closed over rowIsActive and activeGgufVariant before awaiting listGgufVariants, while the same click also starts a status refresh, so the two race and the network call usually loses. The residency test now happens after the await against the settled store, and skips an external checkpoint. adoptResidentModelStatus returned early on an empty status, before the external and modelLoading guards, so an unload from another tab or over the API left the store pinned and the settings page went on treating that row as resident. The empty branch now sits after both guards and clears only a local checkpoint. The monitor's Unload targets the resident local model from /status but cleared the store checkpoint unconditionally, and clearCheckpoint drops the persisted external selection too, so unloading wiped an external pick it never touched. It now clears only when the store holds the model that was unloaded. * Reject booleans for the numeric override fields Pydantic parses non-strictly and bool subclasses int, so a payload with max_seq_length true was stored as 1, a one-token context, and gpu_ids [true] as [1], an unintended GPU pin. _bounded_int already rejects bools for exactly that reason, but never saw one: coercion happens at the route boundary first, which left that guard unreachable through this path. A mode=before validator rejects only booleans, including inside the gpu_ids list, so every other lax conversion still runs and tensor_parallel, remove and fill_absent_fields keep working. * Match both spellings when the monitor unloads a model My earlier fix compared the store checkpoint only against the identifier /status reports, which is the concrete load path. For a GGUF loaded through auto-switch or from a non-active cache the store can hold the advertised repo id instead, so the comparison failed and the store stayed pinned to a model that had just been freed. Match the load path and the public alias, and keep the external guard that prompted the narrower comparison. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Read status before the settings target resolves a quant Two fixes to the API monitor and the Hub settings open. The overlay rearm only cleared `seeded`, so returning from the full page ran the initial-watch seed, and that seed holds running rows back on purpose: at a session's first snapshot such a row started while Studio was loading and nobody has seen it. Off the full page the opposite is true, since that page was showing the same feed with its running rows, so the overlay reopened on exactly the request the user had been reading. The seed now records whether it follows a rearm. Opening a cache row's settings resolves its quant from the store, and the comment there claimed the open kicks a status refresh that lands during the variant lookup. It does not: that refresh comes from an effect keyed on `settingsTarget`, which cannot run until the target exists. The Hub has no polling timer either, re-reading on focus and visibility only, so a window that keeps focus holds a checkpoint from before any API-driven switch for as long as the user sits there, and the editor opens on the quant of whichever model that switch displaced. The handler now reads status itself, alongside the variant lookup. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Clear the derived gguf key on forget and key cached repos by repo id Three fixes. A standalone .gguf is keyed by its bare path, but a load also reads the `<path>:LABEL` entry derived from the filename, which is how the picker keyed the same file before and what the backfill carries over from an upgraded browser. `resolve_model_override_key` cannot fold a bare path onto that spelling, so forgetting cleared a key that was never there and left the real one applying to every later API load, with the settings gone from the UI and nothing left to reach them. The remove branch now clears the derived key too, only for a bare `.gguf` path and only through the resolver, so an ambiguous fold removes nothing. `modelConfigIdentity` decided from the view kind what its own comment says depends on the row. Discover resolves a repo in an inactive HF cache into a discover-kind view whose resource is still the cache row with its snapshot-path run id, so Run from Discover read a key the Downloaded row never writes and loaded with default context, GPU and template, silently. Keyed off the resource instead. The detail view's settings button took the card's quant as given. That quant is a choice only when the user picked it in the selector; otherwise it comes off a store nothing re-reads while the window keeps focus. The card now says which it is, and a derived one defers to a fresh status read. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Consolidate the tests without changing what they prove Test-only. No file under studio/frontend/src or the backend's routes, core, hub, utils or storage is touched. Backend: the override PUT was spelled out at 29 sites as a two-call expression with a local import each time, and 39 tests mocked the store by hand when a fixture does it. A `_put` helper and an `override_store` fixture take both. -186 lines, 273 tests still pass, and the assert count is unchanged at 624. Frontend: eight test files become five plus a shared kit holding the bundler-resolver registration, the localStorage fake and the chat-runtime store fakes that three files had each written out. The resident-status pair merge into one file, and the three identity/storage files into another. 62 tests, 139 assertions, both unchanged. Prose: multi-line docstrings and comment blocks in the test suites keep their opening statement, the rest being recoverable from history. That is most of the remaining reduction, because these tests are close to one line per assertion already. Two consolidations were measured and rejected rather than shipped. A table-driven form of the source-contract tests generates 930 lines to replace 773, since a row costs what an assert line costs. Parametrising the backend key-folding and carry-over families saves nothing once the helper above removes their boilerplate: what is left is the per-case reason, not repetition. Every mutation these tests were written to catch still reddens: reverting new-traffic.ts, adopt-inference-status.ts, settings.py and hub-page.tsx to their pre-fix parents each fails the expected tests. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Resolve a local quant folder's variants by its path A local row carries a repo id only inside the HF cache: getLocalHubId returns null unless the source is hf_cache. A plain folder of quants, under the models dir, a custom folder or LM Studio, therefore has none, while the backend still marks it as needing one, since requires_variant is scan_path.is_dir(), and leaves format_variant null because a directory is not the single-file case. settingsGgufVariantForRow then returns null, the guard is entered, the lookup is skipped for want of an id, and the toast is all that is left. The row menu offers Settings on every non-dataset row, so for these folders it could never do anything. The listing takes a path in that position and scans it, before the repo-id validation that would otherwise reject one, and that is already the request the on-device card makes: its fetch state is keyed on the model id, which for a local row without a hub id is the load path. So both surfaces now choose from the same quant set, and the settings key, the row's load id, does not move. * Stop clearing the checkpoint on an empty status, and forget every spelling Three fixes. An empty /status is not the same as the model going away. With idle unload on, the server frees the resident GGUF and keeps a stash the next request reloads, while get_status never consults that stash and InferenceStatusResponse carries no field that tells the two apart. The store is what names the model meanwhile, for the usage examples and for the reload itself, and usage-examples.tsx says as much in its own words. The chat runtime, which this adoption mirrors, resets only capability flags on an empty status and leaves the checkpoint pinned; every other clearCheckpoint caller is tied to the action that caused the unload. The clear I added two rounds ago is removed, along with the test that pinned it. Forgetting an override cleared only the key a load resolves to. An exact hit short-circuits before the fold, so with two spellings of one model in the map, from a build whose setter stored the literal id, the one named went and the survivor became the sole fold match: the next load applied what had just been forgotten. The folding rule is factored into one helper and the remove branch clears every key under the identity. POSIX paths still stand alone, so two files never clear each other. Both settings handlers now read status before building a target rather than only in the variant branch. Every path out of them seeds the editor from the store and Apply reloads with what it seeded, so a target built on a pre-switch read could reload a resident model from defaults and overwrite its saved config if applied before the read landed. A quant the user picked in the card is still preserved. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Tighten the comments across this branch Collapse multi-line comment blocks to one or two lines and drop comments the code already states. The reasons behind the non-obvious rules are kept, just shorter: the empty-status checkpoint rule, POSIX vs Windows path folding, the legacy path:LABEL override key, reading status before building a settings target, and the rearm seed. Comments only, no code changes. * Never let the monitor's api-key check fail a load _request_used_api_key decides one monitor label, and it already wraps the header read in a try because a caller may hand it anything. The unpack sat outside that guard, so a request object whose headers answer with a non-string reached `scheme, _, token = header.partition(" ")` and raised ValueError out of _load_model_impl. That is exactly how the load routes are driven in tests: a MagicMock request answers every attribute, so `or ""` never fires and the unpack blows up. Twelve tests across test_nvfp4_load_error_message.py, test_validate_gguf_runtime_message.py and test_chat_load_during_training.py failed on this branch and pass on its merge base, which is where this came from. The full backend suite now matches that baseline exactly at 17 pre-existing failures in the same six modules, down from 29. Made the helper total rather than teaching three test modules to build a better double, because a best-effort label must not take a load down for any caller. * Read an empty status against the idle setting, and keep repo quants Two fixes. An empty /status means one of two things and the payload cannot say which. Armed, the idle loop frees the model but keeps a stash the next request reloads, and the store is what names it meanwhile. Disarmed, nothing will bring it back, so the model is gone and leaving the row resident seeds the settings editor from a launch config nothing is running. Neither always-clear nor never-clear is right, and the only endpoint that knows is /openai-auto-switch, which already reports idle_unload_active for the usage examples. The Hub reads it once and the empty-status branch clears only while the loop is disarmed, which is the default. isStandaloneGgufPath tested the .gguf suffix alone. Repo ids ending in .gguf are real on the Hub, an iMat repo among them, and those hold every quant of a model, so reading one as a single file dropped its variant and saved Q4 and Q8 under the same key. The id now also has to name something on this machine: a path anchor, a drive, a second separator, or the bare filename a picked file is echoed back as. A repo id has none of those. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Merge main, and read backend sources as UTF-8 in the contract tests _read_backend arrived with the local MTP drafter work and reads without an encoding, three lines below _read, which passes one. On Windows that decodes as cp1252, and routes/inference.py holds byte 0x81 at offset 103058, so the Windows job died with UnicodeDecodeError before any assertion ran rather than reporting a contract. Same failure on main today; it shows up here because this branch touches the file the helper reads. * Fix six issues the review found, one of them in my own last fix chat_template_override: I gated the re-seed on seedLoadParams and said it implied hydratingExistingModel. It does not, it is !modelLoading, so the re-seed also fired on an ordinary poll and overwrote an unsaved edit on every refresh. It needs both, which is what the neighbouring fields do. The idle-unload setting was read once and cached for the life of the page, so a timeout changed from Settings while the Hub stayed mounted was never seen, and a transient failure stuck on the default forever. It is read with every status read now, keeping the last answer on failure rather than falling back to the default: the default is disarmed, which is the side that clears the checkpoint. Forgetting repo:QUANT cleared the bare repo entry unconditionally, which strips the quants inheriting from it. It is only cleared when no other qualified key survives, so the forget has to be the last word on the model. Provenance would be the better test but nothing records it: the carry-over copies the flags without marking the copy. ApiMonitor.clear dropped an own row that was still running, losing the request outright, since active_count falls to zero and the finish that follows has no entry to land on. The shared pass in the same function already keeps a running row; both follow that rule now. The scoped-clear test used a running row only because start() is the only way to make one, so it finishes the request first and the running rule gets its own test. A cached repo is keyed by its repo id now and used to be keyed by the snapshot path, and resolveInitialConfig has no fallback, so an upgrade read as never remembered. adoptLegacyConfigKey moves the record once, before either entry point reads the new key; a newer save under the new key wins. The server override mirror sent gpu_ids without their namespace. The same integers are Vulkan ordinals under Vulkan and device indices elsewhere, and the server cannot tell, so only a physical pin travels. An absent kind is a record written before the field existed, which was physical only. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix the argument order in adoptLegacyConfigKey, which broke the build savePerModelConfig takes (modelId, ggufVariant, config); the new adoptLegacyConfigKey passed (modelId, config, ggufVariant). tsc caught it, so `npm run typecheck` and `npm run build` both fail at head, and since `npm run build` is the first thing studio setup does, the failure cascades: 30 of 51 checks are red on eb4c3eac8, including startup on all three platforms, the wheel and Tauri builds, Chat UI Tests and every connection job. The runtime effect is worse than the type error suggests. normalize() receives a string, hits its `typeof raw !== "object"` guard and returns an all-defaults config, isDefaultConfig() then reads that as default and takes the delete branch, so nothing is written at all. deletePerModelConfig on the next line still drops the real legacy record, and savePerModelConfig has already returned true. The migration that exists to carry settings across the re-key destroys them instead, and reports success. The contract test asserted the call as a literal source substring, so it passed on the broken code. Corrected alongside, but a substring assertion cannot catch an argument order, so this also adds tests/adopt-legacy-config-key.test.ts, which drives the real storage module and asserts the moved config's values rather than the presence of a key. Two of its four cases fail on the previous head and pass here; the other two cover the paths that never reach savePerModelConfig. Verified: typecheck clean, 103/103 node tests, production build clean, 122 passed across the three tests/studio contract files, and 316 passed across test_openai_auto_switch.py and test_api_monitor.py. * Take the cosmetic churn back out of the test diff The test diff carried 90 pre-existing functions that this branch only reworded or re-wrapped comments in. That is what made the three test files read as 2446 lines of change when the behavioural part is far smaller, and it is the reason the deletion count looked like coverage had been removed: 342 of the deletions were comments being reflowed to a wider budget, not assertions going away. Restored those functions to their merge-base text wherever the change was provably cosmetic. Two rules decided it: comments never reach the AST, so they drop out for free, and a docstring only counts as cosmetic when its word sequence is unchanged, so the 16 docstrings this branch genuinely rewrote are left alone. The executable AST of all three files is byte-identical to the previous head. Also pointed the 19 backend source reads the new tests had inlined at the _read_backend helper the file already owns, which is what the pre-existing tests in it use. Seven of them spanned three physical lines to repeat a path and an encoding argument. The two inlined reads inside pre-existing tests are left as they are, since rewriting those would add diff rather than remove it. No test and no assertion was removed: 278/633, 107/250 and 102/547 before and after, per file. The three files go from 2446 to 2094 lines of diff, and the PR from 9854 to 9502. Verified: 2141 passed and 3 skipped across tests/studio, 486 passed across test_openai_auto_switch.py, test_api_monitor.py, test_openai_auto_download.py and test_parallel_slots_per_load.py. * Fix three ways a saved per-model config is lost or ignored Three review findings, each a case where the settings a user saved stop being the settings a load applies. routes/settings.py, utils/openai_auto_switch_settings.py. A repo cached outside the active HF cache has two spellings: the snapshot path it loads from, and its repo id. The one-time backfill mirrors whichever one localStorage held, a later Settings save writes the other, and nothing retired the first, so the override map ended up holding both. The load ladder reads the path before the advertised id, so every API auto-load kept applying the pre-migration config and the save looked like it had done nothing. Reordering the ladder would only move the bug: an inactive-cache repo is sometimes legitimately keyed by its path, when inventory-dedupe keeps no cached row for it. So a save or a forget now retires the other spelling of the same (repo, quant), which is the one-entry-per-model invariant this route already enforces for casing folds, the <path>:LABEL legacy key and the bare repo id. The retired entry hands over its llama_extra_args first, since the page can neither show nor restore them. The one-time fill retires nothing, or the migration would delete what it is meant to carry. per-model-config.ts. adoptLegacyConfigKey saved under the new id before deleting the old one, so at the entry or byte budget the map briefly held both copies and the save evicted the oldest unrelated model to fit: silently, still reporting success, and with no eviction list to hand back, so that model's server override kept applying with nothing in the UI able to forget it. The key is renamed in one write now, which cannot grow the entry count. Reachable well before the entry cap via the byte cap: 17 models each carrying a 60 KB chat template sit under 1 MiB. apply-inference-status-to-store.ts. An API auto-switch hands the loader the concrete snapshot path, so /status reports that path while the model's settings are keyed by its repo id. Reading remembered slots by the raw identifier missed the record, cleared nParallel on the model change, and the next save wrote the blank back over both localStorage and the server entry. Resolved through the cached-repo alias now, and only for namespaced ids: a file stem two models can share must never be read, or one model's settings apply to another. Two further findings are not changed here, with the reasoning in the PR thread: the duplicate-key collapse cannot affect a load, because the override keys come from the resolver and never from the request's spelling; and preserving a bare repo entry for a quant that inherits it is unsatisfiable alongside test_forget_clears_the_bare_repo_entry_the_quant_inherited_from, since neither case leaves anything in storage to tell them apart. Verified: 492 passed across the four backend suites this touches, 2142 passed and 3 skipped across tests/studio, 118 node tests, typecheck, build and ruff clean. Each fix was confirmed against its own tests on the unfixed source first: 4 fail for the alias retirement, 2 for the eviction, 5 for the slot lookup, and the two over-deletion guards pass either way. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Tighten the comments this branch added Same reasons, fewer lines: collapse the multi-line explanations added by this branch and drop three field docs that only restated the field name. No code, docstring or test changes. * Stop a lower-priority override key holding newer data than the one a load reads Four defects, all one shape: a write leaves a key the loader reads before, or instead of, the key holding the values the user actually meant. Standalone GGUF saves now consult the filename-derived legacy key. The loader reads the bare path ahead of "<path>:LABEL", and the settings page never sends llama_extra_args, so the backend carry-over was the only thing preserving those flags; for a colon-free path neither _bare_model_id nor cached_repo_alias_keys fires, so the new bare entry won and the flags became unreachable. The removal branch already called _legacy_standalone_gguf_key; the save branch now does too. The one-time fill no longer creates a snapshot-path key over a repo-id entry. resolve_model_override_key cannot fold a path onto a repo id, so a backfill from a pre-upgrade browser wrote a brand-new higher-priority key and skipped alias retirement, shadowing newer server values on every API load. The redirect is direction-aware: only the path spelling outranks, so a repo-id fill over an existing path entry still creates its own key, and two snapshot paths never redirect since neither is knowably the load path. Carried pass-through flags no longer shadow first-class controls. A save keeps stored llama_extra_args while writing the field just edited, and the auto-switch mapper sent both, so a stale "--ctx-size 8192" landed after Unsloth's own flags and won llama.cpp's last-wins parse against a freshly saved 32768. The /load route already strips exactly these groups off inherited extras; that stripper is imported rather than mirrored, gated per group on the field the override supplies, so a flag with no first-class field behind it still passes through. A same-model reload now advances the chat template. hydratingExistingModel is computed from checkpoint and quant alone, so a reload from another client left the Hub presenting the old template as live, and the next Apply persisted and reloaded it over the new one. The baseline always advances; the control advances only while it still equals the old baseline, so a mid-edit value survives. Blank and null compare equal, matching cleanTemplate and the backend normalisation. The resolver is a separate module because the applier imports the model-picker barrel, which is unimportable under node --experimental-strip-types. Each fix was checked against its own test on the unfixed source first: 2 fail for the settings-key pair, 2 for the shadowing flags, 5 for the template seed. 326 backend, 316 llama-server/chat-load, 103 contract and 127 node tests pass, with typecheck, build and ruff clean. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Stop the monitor acting on a snapshot the API has already moved past Two defects on the new monitor, both where this PR's own feature is the concurrent actor: an API auto-switch runs while the page is watching it. Turning automatic opening off no longer arms a delayed pop. The poll stood down without re-arming the watch, so API-key calls landing during the opt-out stayed unseen and the first snapshot after re-enabling reopened the panel for an hour-old burst. The suppression guard could not stop it either, since lastNewEntryAt is frozen for the whole opt-out and quietFor always exceeds REARM_QUIET_MS. That contradicts the invariant stated a few lines above it, that a backlog built while the poll stood down is not new traffic, which the other stand-down already enforces. Re-arming alone is not enough: autoOpen is an observer dependency, so the re-enable render folds the stale pre-opt-out snapshot first and spends the re-arm before any fresh poll resolves. observeResponse now folds each snapshot at most once, and standDownWatch re-arms only a watch that has seeded, so a session that merely starts opted out keeps its first-snapshot semantics. Unload no longer reports success over a model it did not free. The button named the checkpoint it read from /status, but /unload takes the lifecycle gate the auto-switch holds across a whole swap, and the switch does substantial awaited work before teardown, so status answers with the outgoing model for a wide window. Naming a model a load has replaced is a 200 no-op, so the button reported success while the new model kept its VRAM. There is no unload-current route, so unloadResident reads, unloads and re-reads, retrying once and surfacing the model still resident instead of swallowing it. Steady-state cost is one extra GET /status per click and never an extra unload. Note on the contract test this moved. test_monitor_unload_clears_only_the_model _it_freed asserted a verbatim source line, so it passed on the racy single-pass unload and broke only when the text moved; it never covered its own name. The assertion now matches the new shape and its docstring points at the node test that exercises the race. Each fix failed its own test on the unfixed source first: 2 for the re-arm, of which one pins that the naive one-line remedy is still broken, and 3 for the unload. 2142 tests/studio, 326 backend, 137 node tests pass, with typecheck and build clean. * Release a settings open on the refresh that superseded it, and report a refused clear A superseded status read no longer releases the caller awaiting it. The drop branch was a plain generation check that wrote nothing and still resolved, and settingsOpenSeq is bumped only by opening settings, so a focus or visibility refresh could supersede a settings-open read while passing that guard. Every other superseder is either covered by settingsOpenSeq or starts after the awaited read and therefore wins; focus and visibility are the only uncoordinated ones. Half of the reported harm does not occur: the editor cannot keep the old model's config, because loadedConfigSignature is folded into the React key so the view remounts when the live config lands, and setting settingsTarget fires a fresh refresh that always starts last. What does not self-heal is the quant baked into settingsTarget by the stale read, which is what runSettingsTarget passes to selectModel and what settings are then persisted under. That is the harm test_detail_settings_defers_a_derived_quant_to_a_fresh_status_read already guards elsewhere, so a dropped response now resolves with the refresh that superseded it. Only a strictly newer sequence hands back a promise, so the unmount cleanup, which bumps the sequence without starting a read, cannot make the last refresh await itself. A refused clear no longer fails silently. clearApiMonitor rejects on a network error and on any non-ok DELETE, clear() had no catch, and the page discards the promise with void, so the detail pane emptied while the log stayed put with no message and an unhandled rejection behind it. The hook already owns an error state the banner renders, so the failure routes through it and the reload is skipped, since nothing was deleted. One limit worth stating: when the DELETE alone fails while polling still succeeds, the next poll clears the shared banner within about a second, so the message flashes. That is inherent to reusing the existing error state, and it still beats silence. The case it genuinely fixes is a paused monitor, where the poll returns early while the button stays enabled, so the failure was permanent. Each fix failed its own test on the unfixed source first: 2 of 6 for the supersession, 2 of 3 for the clear. 2142 tests/studio and 146 node tests pass, typecheck and build clean, and no contract assertion needed changing. * Serialize override writes so two spellings cannot retire each other The override save handler is a plain def, so FastAPI runs it in a threadpool and two requests really do interleave. Each individual write is atomic, since set_model_override goes through upsert_app_setting_map_entry under BEGIN IMMEDIATE, but nothing spans calls: the target write, the alias read and the alias delete are three separate transactions, and the only lock in either module guards the two second memo. So when one client saves a cached quant by repo id while another saves it by snapshot path, and both writes land before either cleanup, the repo-id request deletes the path row and the path request then deletes the repo row. Both calls return 200 and the store ends empty. The two spellings are not hypothetical. The shipped UI keys a cached repo by repo id, but the path spelling is why cached_repo_alias_keys exists at all, the browser's write queue is keyed by spelling so the two go out in independent queues rather than serialized, and the eviction mirror sends dropped.modelId straight from localStorage, which can still be a legacy snapshot-path key because the openModelSettings branch never adopts it. It is also a documented REST endpoint any client can call. The server is a single uvicorn process with no workers configured, so a module level lock is a complete fix. It is applied as a decorator rather than a with block for two reasons: the handler body is not reindented, which keeps the verbatim-source contract assertions intact, and the remove branch's four-write sequence is covered by the same lock. The new test pins both threads just after their target write with a barrier, falling back on a broken barrier once the writes serialize so the fixed path cannot deadlock, and locks the fake store per entry to mirror BEGIN IMMEDIATE so only the gap between calls is on trial. It fails on the unfixed source with an empty store. 327 backend, 103 contract and 160 adjacent settings-route tests pass, ruff clean, and no contract assertion needed changing. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: Unsloth <michaelhan@Michaels-MacBook-Pro.local> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: danielhanchen <danielhanchen@gmail.com>	2026-07-31 04:24:17 -07:00
Nilay	e34c5090b0	Desktop: open uploaded files via the OS instead of window.open (#7660) ... * Desktop: open uploaded files via the OS instead of window.open * Preserve compare audio metadata * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: Wasim Yousef Said <wasimysdev@gmail.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-07-31 11:01:31 +02:00
Michael Han	a7031d603f	Studio: profile usage stats (#7593) ... * Studio: add profile usage stats and tidy the personalization panel Adds a "Your stats" section to the Profile settings tab, built entirely from local history in studio.db. No telemetry, nothing uploaded. Backend - storage/profile_stats_db.py folds every metric in one streaming pass over chat_messages, memoised against a (count, max created_at) fingerprint so reopening the tab is free until history changes. - routes/profile_stats.py serves GET /api/profile/stats from a worker thread, so a cold pass cannot stall token streaming. Frontend - Headline tiles, a token activity grid with daily/weekly/cumulative modes, activity insights, most used models, hour and weekday rhythms, and training run totals. - The stats panel is lazy loaded so recharts stays out of the main bundle. - Personalization panel now puts a larger avatar beside the two name fields, with picture options moved into an edit popover. - "Sloth in greeting" moves to Chat defaults, next to the other chat toggles. Tests: 9 backend tests including a regression test that the endpoint does not block the event loop, plus formatting unit tests. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: correct profile stat aggregation Six accuracy fixes, each with a test that fails without it. - firstTokenTime is already an elapsed duration, not a timestamp. Subtracting streamStartTime made the comparison false for every real message, so "Average time to first token" was always empty. - Forking clones the whole ancestry into the new thread with the original timestamps. Those copies were counted again, doubling tokens, messages, attachments and activity for the source conversation. Rows older than the fork are now skipped. - training_metrics.num_tokens is state.num_input_tokens_seen, a running total logged at each step, so summing the samples multiplied the real figure. Take each run's final counter, matching get_run_metrics. - A resumed run continues its source's step and token counters from the checkpoint, so adding both reported the same progress twice. Only runs no later run resumed from are counted. - Days, hours and weekdays were bucketed in the server's timezone while the client parses the keys as browser-local. The endpoint now takes the caller's getTimezoneOffset(). - A turn with no contextUsage.modelId fell back to the thread's model_id, which tracks the current selection and so misattributed older turns after a mid-conversation switch. Those turns are left uncredited instead. * Studio: further profile stat corrections Follow-up to the previous pass, each with a test that fails without it. - Cancelling a run also sets resume_blocked, so filtering on that flag alone dropped the steps and tokens of cancelled work while still counting the run and its duration. Only a run a later run resumed from is excluded now: the resume claim leaves output_dir intact, cancelling clears it, which tells the two apart. - A fixed getTimezoneOffset() was applied to every historical message, so winter records read an hour out when the panel is opened on summer time, moving messages near midnight onto the wrong day. The endpoint now takes an IANA timezone and converts each timestamp with its own offset, keeping the fixed offset as the fallback. - A fork with no new turn was skipped before its thread was registered, so it did not appear in the chat total until its first message. The thread is now counted before the cloned rows are suppressed. - Routed and aliased responses record the model that actually answered in responseDetails.responseModelId, while contextUsage.modelId stays the requested checkpoint. Most used models now prefers the former. - Renamed runs showed the model label instead of the name the user chose. The name leads and the model moves beside the dataset. * Studio: harden profile stats against deleted and malformed history - Deleting a forked conversation's source left the fork holding the only copies of those messages, but the skip still dropped them because forked_from_thread_id is not a foreign key. The suppression now joins the source row, so copies are only ignored while the originals survive. - delete_run never clears the predecessor's resume_blocked, so removing a continuation stranded its source at zero steps and tokens while the run stayed visible. Supersession now requires a continuation that still exists. - created_at is a client-supplied integer stored unchecked, so one out-of-range row made datetime raise and returned 500 for the whole panel. Those rows are now skipped for day and hour buckets. - A future-dated row satisfied the current-streak window, reporting days that have not happened. The last active day must now be today or yesterday. - Compact numbers rounded 999,999 to "1000K". Rounding into the next unit now steps up the suffix. - Escape or a programmatic close unmounts the Profile tab without a blur, dropping an in-progress name edit. Drafts are committed on unmount; both saves no-op when unchanged. - Stats copy no longer claims the numbers never leave the device, which is untrue when Studio is reached from another machine. It states what actually holds: nothing is collected or sent to Unsloth. * Studio: dedupe sibling forks and group comparison panes - Deleting a thread that had been forked more than once left every sibling holding a copy of the shared ancestry, and the source-existence gate then let all of them count it, multiplying the usage by the number of forks. One fork per dead source is now elected to keep its copies, the one carrying the most, so nothing is lost or counted twice. - Compare mode persists one thread per pane under a shared pair_id and the sidebar renders them as a single conversation. The chat total now keys on the pair, so one comparison is one chat and average tokens per chat is not halved. - Corrected the note on cumulative mode: it is the running total across the displayed window, not lifetime. Seeding it with everything older than the window would flatten every bar against a baseline the grid has no room to show. * Studio: count fork clones per message and drop future streaks - A pre-fork message can be pruned from its original thread while the clone stays in the fork. The suppression only checked that the source thread existed, so that message's tokens and activity vanished. Clones are now matched to the row they came from, and one fork per source is elected to stand in whenever the original is gone, whether the whole thread or just that message went. - Future-dated history was filtered out of the current streak but still padded the longest streak and could be reported as the last active day. It is dropped before any streak field is computed. - Cumulative mode labelled its tooltip "week of" while showing the running total. It now reports that week's tokens; the bar height still uses the running total. - The activity grid and weekday axis formatted dates with the browser language rather than the language chosen in Settings, and the memos could not react to a change. They follow the app locale now. The fork fixtures were also corrected: fork_chat_thread copies created_at verbatim, so a clone carries the original's timestamp, which the previous fixtures did not model. * Studio: drop the hour and weekday rhythm cards Removes the "When you work" and "Your week" row from the Profile stats. The backend stopped computing the hour and weekday buckets too, since nothing else read them, and the payload no longer carries them. The timezone and malformed-timestamp tests now assert on day buckets, which is what the activity grid and streaks actually use; the daylight-saving case moved to a timestamp where the hour of drift crosses midnight so the day still proves it. This was the only recharts consumer in the profile panel, so the lazy split's note about keeping charts out of the main bundle no longer applies and has been corrected. Build drops about 22 KB across two fewer chunks. * Studio: harden the profile stats aggregation Four fixes from review: - _as_float raised OverflowError on a JSON integer wider than float, so one oversized counter returned 500 for the whole panel. It degrades to zero now, like every other unreadable field. - Fork dedup elected one winner per source thread, but fork_chat_thread copies a single parent_id branch, so sibling forks of a retry and a regeneration hold different rows. Electing per original message keeps each branch when the source is deleted. - create_run claims a resume source before the continuation logs its first step, so a continuation that failed early took the source's completed steps and tokens with it. Supersession now needs the continuation to have reached the source's step. - Cumulative activity is a running total over the displayed window, but a narrow card trims older weeks without rebasing, opening the first visible bar at the hidden total and flattening the rest. * Studio: record resume lineage and tighten the stats fallbacks Four more from review: - peakDay and activeDays read every day bucket, so a future-dated message could headline a date the activity grid cannot show. Both now stop at today, like the streaks and the grid already did. - Supersession matched runs by output_dir, but cancelling a resumed continuation nulls that column, which un-superseded the source and double-counted their shared steps and tokens. training_runs now stores resumed_from_run_id, which create_run already accepted and discarded. Rows written before the column stay on the output_dir heuristic. - Token totals read contextUsage only, while the response details sheet already falls back to serverTimings prompt_n, predicted_n and cache_n. Local-engine replies were undercounted against what the app displays. - The stats section was missing from the settings search index, so the headline labels were unreachable from search. * Studio: close profile stats review gaps --------- Co-authored-by: Unsloth <michaelhan@Michaels-MacBook-Pro.local> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: oobabooga <112222186+oobabooga@users.noreply.github.com>	2026-07-29 08:02:14 -07:00
alkinun	502730bbba	Studio: add Deep Research (#7219) ... * Studio: add durable Deep Research workflows * Studio: preserve research integration after upstream updates * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: keep research worker compatible with Python 3.11 * Studio: address Deep Research lifecycle review * Studio: preserve durable research recovery * Studio: preserve research stream and context * Studio: harden research sources and limits * Studio: align research with shared chats * Studio: guard durable research actions * Studio: protect durable research turns * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: deepen durable research decisions * Studio: protect research prompts and queries * Studio: slim research stream deltas * Studio: preserve research evidence and citations * Studio: harden Deep Research (CI, prompt injection, query PII, config, citations) - Fix backend CI: add research_runs_router to the synthetic routes stub in test_desktop_auth so studio.backend.main imports under the health-check test. - Escape prompt-delimiter tags in the decision and synthesis prompts so gathered web/document content cannot close an <untrusted_...> wrapper and inject instructions into the local planner/decision/synthesis model. - Extend the public-query sanitizer to redact Luhn-valid payment cards, phone numbers, non-global IPs, and labeled private identifiers before a query can reach web search. - Reject nested credential keys in inferenceRequest and ragScope, not just top-level keys, when persisting a durable run config. - Treat maxSources as one budget shared across web and document sources (collection and resume paths) instead of per type, which allowed up to 2x the configured cap. - Preserve document citations whose filename contains a closing bracket by tokenizing valid citations before stripping invalid ones. - Persist Deep Research off when switching to an external model and when enabling Web Fetch so a refresh cannot rehydrate a mutually-exclusive state. - Add regression tests for the query, prompt, citation, and config hardening. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: make the research claims table migration atomic The owner-scoped to global claims migration ran its RENAME, CREATE, INSERT and DROP in autocommit, so an interruption after CREATE left the new table empty, orphaned the rows in the legacy table, and never re-triggered. Wrap the rebuild in an explicit transaction so a crash rolls back cleanly and the migration re-runs on the next boot. * Studio: block message edits and regeneration during an active research run After a reload a durable research run is followed by the research store rather than an assistant-ui run, so thread.isRunning is false while research is still active. Message edit, refresh and the edit composer previously gated only on isRunning, which let a normal generation start alongside the running research run. Gate them on the active thread's research state as well. * Studio: keep the plan review mounted through approval Keying PlanReview on planRevision remounted it mid-approve when updateResearchPlan bumped the revision, resetting the local pending flag and re-enabling Start research while the approve was still in flight, which allowed a duplicate approve. Key on runId only. * Studio: drop the redundant deep-research persistence change setCheckpoint already persists Deep Research off for external models at the top of the function, so the added saveBool was a duplicate, and clearing Deep Research from setWebFetchToolsEnabled guarded a state that is not reachable (Deep Research is local-model only while the Web Fetch pill is external-provider only). Revert both to the pre-hardening version. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: harden Deep Research citations, query privacy, and message protection Address review findings in the Deep Research backend: - Escape an unbalanced ")" in citation destinations so a source URL cannot close the markdown link early and inject a second link, keeping balanced parentheses literal. - Match raw-URL citations on whole tokens so a URL sharing another URL's prefix is no longer partially rewritten. - Redact non-global IPv6 addresses in public search queries, matching the existing IPv4 handling. - Detect credential key names after normalizing case and separators so nested openaiApiKey, accessToken, and clientSecret values cannot be persisted. - Reject client edits to server-managed research prompts and reports at the storage layer; only the internal writers pass allow_research_update. - Scope research searches to the first allowed domains instead of dropping site scoping for large allow lists. - Persist the same fetch evidence bound used during live synthesis so a resumed run is not shortened. - Scope run completion so it only replaces this run's message parts. Add regression tests for the above. * Studio: fix Deep Research SSE framing, source counts, and favicon privacy - Normalize the whole SSE buffer so a CRLF split across transport chunks still frames events. - Count web and document sources together in the activity header so a RAG-only run is not shown as zero sources. - Cap the plan editor at the run's configured maxSteps instead of a hard-coded 30. - Add an allowRemoteIcons opt-out to the sources components and disable third-party favicon requests for research sources so visited domains are not leaked. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: address final Deep Research review findings * Studio: fit Deep Research synthesis evidence to loaded context, add opt-in web grounding Size the synthesis evidence budget to the loaded model context so the prompt is not silently truncated on small contexts. When the evidence overflowed the window the report degenerated (it echoed the evidence tail instead of writing); the budget now reserves tokens for the prompt scaffolding and converts the remainder to chars, keeping the full cap when the context is unknown. Add opt-in web grounding for auto-read: read the top search results, ingest them into an ephemeral RAG scope, hybrid-retrieve the passages most relevant to the question with the existing knowledge-base retriever, and fold those chunks into the step evidence. The scope is per call and deleted afterwards, so a user's knowledge base is never touched. Off by default; enable with UNSLOTH_RESEARCH_AUTO_SCRAPE=1. Gated per run by budgets["maxAutoScrape"], so runs created without it keep legacy snippet-only behavior, and grounding is skipped when the loaded context is too small for the prompt. Add tests for the adaptive evidence budget, scraped-text cleaning, the ephemeral web-RAG retrieval and scope cleanup, and the auto-read evidence path. * Studio: read Deep Research synthesis context from the inference orchestrator Make the adaptive synthesis-evidence budget actually engage in the normal Studio architecture. _loaded_context_length read core.inference.inference, the low-level backend that lives in the model subprocess and stays unpopulated in the main web process where the research supervisor runs, so it returned None and the budget silently fell back to the 32000 character cap (leaving the report exposed to the truncation this was meant to fix). Read the inference orchestrator instead, and the llama.cpp backend for GGUF, mirroring routes.inference._monitor_context_length so the budget sizes to the context the API layer serves. Verified on a running server: at a 12288 token load the probe now reports 12288 and the budget adapts to 24576 characters instead of the 32000 fallback. Also: - Reserve context for the generated report as well as the prompt scaffolding (raise the reserve to 4096 tokens) so evidence does not crowd out the output on a small window. - Honor a numeric UNSLOTH_RESEARCH_AUTO_SCRAPE by passing the per-run maxAutoScrape as the page cap to the scraper, instead of always reading the maximum. - Guard the web-RAG connection acquisition so a get_connection failure returns the documented empty result rather than propagating. - Add a synthesis-context test that patches the real backend accessor (not the probe itself) so the production wiring is exercised, plus a scrape page-cap test. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: harden Deep Research query redaction and research autosave - research_runs: extend the opaque-token allowlist so unlabeled Hugging Face (hf_) and GitLab (glpat-) tokens are redacted before a query can reach web search, without over-redacting public model or version ids. - runtime-provider: for a server-managed research message, echo the backend-stored metadata verbatim on autosave. Merging the client metadata re-added client-only fields the server never persisted, so the server-side guard saw a diff and rejected every streamed or snapshot update with 409. * Studio: keep composer tool pills always accessible after merge The merge left the composer line marked always-expanded (data-expanded "true") while the inner pill row was still gated behind composerExpanded, so the Search and Code toggles disappeared once the permission mode was "off" with no other toggle set. Render the primary tool pills unconditionally, matching the always-expanded layout, and drop the now unused composerExpanded and permissionMode locals. Fixes the Chat UI Playwright check that asserts the Search and Code pills stay visible. * Studio: update Deep Research composer contract to always-expanded layout The always-expanded composer no longer routes effectiveDeepResearchEnabled through a composerExpanded expression, so the frontend contract now checks that it gates the Deep Research composer button render instead. * Studio: do not bind a research run to a populated assistant reply create_run adopted any assistant message under the user turn whose researchRunId was unset, including a prior answer reused by a retry. On completion _update_assistant drops the untagged text and source parts, so that answer was silently overwritten. Only bind to an empty placeholder or this run's own message, and reject a reply that already carries content. * Studio: harden Deep Research synthesis budget, prompt shielding, and message protection - research_runs: split the synthesis evidence budget evenly across notes so a small context still keeps a slice of every research step instead of dropping the later steps after the earliest ones fill the budget. - research_runs: shield the research question and approved plan before placing them in the decision and synthesis prompts, so a closing delimiter in either cannot escape its block and inject sibling sections. - research_runs: redact bearer authorization tokens from public search queries. - studio_db: include attachments in the research-message change check and guard direct attachment deletion, so server-managed research prompts and responses cannot be mutated through the attachment paths. - chat_history: map the protected-message conflict on attachment deletion to 409. * Studio: strip invalid document citations that contain brackets The invalid-citation regex stopped at the first closing bracket, so a citation whose filename contained brackets left its tail (".pdf, p. 9]") in the report. Match a balanced bracketed span so the whole invalid citation is removed; valid citations stay protected by the earlier tokenization pass. * Studio: free the RAG search slot when a lookup times out or is cancelled The bounded knowledge-base search held the sole admission slot in a detached worker until the search returned, so a lookup that outlived its timeout (a stalled embedding or blocked vector call) kept the slot forever and starved every later lookup, disabling knowledge-base retrieval globally. Release the slot from the caller when it stops waiting, exactly once, so a detached worker finishes without re-holding it. * Studio: remove Websites label from research composer * Studio: fix Deep Research review findings (RAG slot bound, orphaned workers, hardening) - Bound the shared RAG search slot to one running worker. The search that is doing the embedding/index/GPU work now owns the admission slot until it finishes, instead of freeing it on caller timeout while the detached worker keeps running, which let a second search enter and stack concurrent work behind the capacity-of-one semaphore. - Cancel active research runs before deleting their thread, project, or all history. Deleting cascade-drops the run row, but the worker only notices at its next lease check, so it could keep doing model/web/RAG work for a run that no longer exists; signalling cancel first shortens that window. - Shield the planner prompt's conversation and question with _shield_untrusted, matching the decision and synthesis prompts, so untrusted text cannot forge planner delimiters. - Do not let a research key-revocation failure replace a successful non-streaming completion; log it like the streaming path does. - Include created_at in the protected research-message guard so a client cannot reorder server-managed prompt/response messages while leaving the body intact. - Reject non-scalar ragScope values; a nested container evades the sensitive-key scan when its inner keys are unlisted and would reach retrieval code that expects a scalar scope id. Adds regression tests for each. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: remove research composer globe icon * Studio: use Hugeicons telescope in research composer * Studio: use Telescope02 icon in research composer * Studio: standardize Deep Research telescope icons * Studio: move Deep Research below web and code tools * Studio: merge grounded page excerpts with search snippets instead of replacing When auto-scrape grounding retrieved page-body chunks, it replaced the raw search-result text for that step. If the retrieved chunk was a distractor or dropped the key fact, the answer-bearing search snippet was lost and grounded runs regressed below snippet-only accuracy on factual questions (e.g. returning Apache 2.0 instead of the Qwen License, 403 instead of 404, or a single mirror diameter instead of the sum). Keep the search snippets and append the grounded excerpts as supplementary evidence via a small _merge_scraped_evidence helper. Grounding stays opt-in and off by default, so legacy runs are unchanged. Adds regression tests. * Studio: fix stale website access assertion in Deep Research contract test The dialog heading was renamed to a DialogTitle, so the contract test still asserted a <span>Websites</span> that no longer exists and failed on every branch built on this one. Assert the current heading instead. * Add AGPL-3.0 SPDX header to the two new test files for PR #7219 * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix citation loss, effort clamping and nested inferenceRequest for PR #7219 Three review findings, each with a regression test that fails without the fix. Citation dropped for a bare URL in prose parentheses. _RAW_URL swallows the closing paren and the old trim set only stripped ".,;:!?", so the catalog lookup missed and the validator deleted the whole citation, leaving an unbalanced "(" in the report. New _trim_url_tail follows GFM extended autolink path validation: one right-to-left pass that interleaves punctuation and unmatched-")" trimming. Both rules must run in the same loop, else "https://x/y.)" keeps a stray dot. Balanced parens inside a URL (Wikipedia-style) still survive. Output verified against cmark-gfm on nine cases, including "https://x/foo)bar)" which must keep ")bar". Research runs forwarded reasoningEffort unclamped. The local chat path clamps to the loaded model's advertised levels; the research branch did not, and the backend only validates enum membership, so llama.cpp dropped a level the model lacks and the whole durable run silently fell back to the template default. Now uses the same helper and the same levels as normal chat. Note this makes "max" on a gpt-oss low|medium|high model resolve to "low" rather than falling through to the template default, matching normal chat exactly; the divergence between the two paths was the bug. Nested inferenceRequest values were persisted. Every allowed field is a scalar and the numeric/bool/enum ones reject a container while coercing, but "model" is stringified with str(), which never raises, so {"auth": "sk-..."} slipped past the sensitive-key scan ("auth" is not on the list) into the durable run config as the model id. Mirrors the ragScope guard already in this PR. Verified: 542 passed across the research/web/sandbox/chat-history backend suites, frontend contract 10 passed, tsc --noEmit clean. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix report-stalling regex, uncataloged KB evidence and bracketed titles for PR #7219 Catastrophic backtracking in _DOCUMENT_CITATION. The alternation (?:[^\[\]]+|\[[^\[\]]*\])* backtracks exponentially on an unterminated "[Document:" with no later bare "]", which is ordinary malformed model output and exactly what this sanitizer exists to handle. Runtime quadrupled every two characters; one realistic 76-char line did not finish in 90s. It runs synchronously inside async _research (the line below it uses asyncio.to_thread), so a single bad report pins the event loop and stalls all of Studio, not just the run. Replaced with the language-equivalent unrolled form, verified identical on well-formed inputs including bracketed filenames, and linear: a 20,000-char tail now takes 0.4ms. Not using possessive quantifiers or atomic groups, which need Python 3.11 while this package declares >=3.9. Uncataloged knowledge base evidence reached synthesis. When maxSources is already full, every returned chunk hits the continue, so accepted_rag_sources stays empty, the "if accepted_rag_sources" rebuild no-ops and rag_result keeps the raw KB text. That text has no document_source_catalog entry, so the validator strips any citation to it and synthesis is left building claims on private KB chunks it cannot attribute. Cleared, gated on rag_sources so a text-only KB reply is still passed through. The resume branch built rag_evidence from all restored sources with the same hole, so it now mirrors the live loop. Bracketed source titles destroyed their own citation. The catalog gave the model the raw title while the citation writer stripped brackets. Search titles routinely carry one ("[PDF] Annual Report"), and the prompt tells the model to copy the title verbatim, producing a label the validator cannot match. Both sides now share _citation_title. Verified: 756 passed across the research/web/sandbox/chat-history/rag backend suites. Each fix has a regression test that fails without it. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Keep a durable run alive when no model is loaded for PR #7219 A durable run is claimable within the supervisor's poll interval of startup (main.py starts it in the lifespan, and claim_next takes any 'running' run whose lease expired), Studio has no startup model auto-load, and the browser is not connected yet. So restarting Studio mid-run reliably lands the next model call on the local endpoint's HTTP 400 "No model loaded". That 400 is not retryable: _completion retries only >= 500, and _stream_completion, which serves both planning and synthesis, has no retry at all. The run is marked failed, and the only recovery is retry, which sets report_text NULL and deletes every research_plan_step, research_source and research_document_source. Up to an hour of scraping and synthesis is lost on a plain restart, on the feature whose whole point is surviving one. Treat only that refusal as transient: wait up to the run's own modelTimeoutSeconds for a model to come back, then re-send. Any other 400 still fails immediately, so no behaviour changes on the happy path. The wait polls _check_active, so cancellation and lease loss are still honoured, and the model probe fails open, so a probe error can only send a request, never withhold one. Each wait is bounded by the run timeout and the number of waits per call is capped, so a model that keeps disappearing cannot re-send forever. Deliberately not pinning or restoring the model, which the review comment also suggested. Auto-switch is opt-in, default off, and GGUF-only, so restoring would silently evict the model the user just loaded from a background worker, and comparing the configured name to the loaded id is fragile across variant suffixes and advertised aliases, so it would break working runs. Verified: 853 passed across the research/web/sandbox/chat-history/rag/inference backend suites. Eight of the nine new tests fail without the fix. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Make website-policy search reach the whole allowlist and refill past blocks for PR #7219 Two review findings on the website access policy. Domains past the site: filter cap were undiscoverable. The policy accepts up to 100 allowed domains and the prompt tells the model all of them are searchable, but scope_search_query always scoped to allowed[:8], so a source in the ninth or later domain could never be found, and an undiscovered URL cannot be fetched either. The cap itself is right, search engines stop honouring long OR chains, so the window now rotates by a hash of the query instead of being a fixed head. Every allowed domain is reachable across a multi-step run, the same query is always scoped the same way, and lists at or under the cap are unchanged. A page of blocked results returned nothing. The policy filters after the search while DDGS was asked for exactly max_results candidates, so if those happened to be disallowed the tool reported no results even when valid ones ranked just below, wasting a research step. Ask for a deeper pool when a policy is set and stop at max_results allowed entries. No policy means no over-fetch, so ordinary searches are unchanged. Verified: 2324 passed across the research/web/sandbox/chat-history/rag/tool backend suites. The 8 test_studio_api.py failures are pre-existing and need live OpenAI/Anthropic credentials; they fail identically with these changes stashed. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Only overfetch search results when the website policy restricts for PR #7219 Follow-up to 8be0b3699. Every run stores normalize_website_policy(...), which returns {"allowedDomains": [], "blockedDomains": []} and is truthy even when nothing is restricted, so the default unrestricted path asked DDGS for four times as many results on every step. That is pure added latency and timeout risk, since the filter passes everything and only max_results entries are returned either way. Test the domain lists rather than the dict. * Budget the whole research prompt against the loaded context for PR #7219 Only the synthesis evidence was budgeted, so the budget could not prevent the overflow it existed to prevent. Measured at head with a realistic prompt (40-source catalog, 12-step plan): the untrimmable scaffolding is about 7,900 chars and the conversation context adds up to 12,000 more. On a 4096-token context, which is the GGUF auto-fit floor and the transformers default, the synthesis request came to about 1.7x the window. Worse, _synthesis_evidence_budget computed usable_tokens = 0 at or below the 4,096-token reserve and then returned the 1,500-char floor anyway, so it added evidence to a prompt that already did not fit. The decision prompt had no context awareness at all: a fixed evidence[-60000:], roughly ten times a small window, on every step rather than once at the end. Overflow is not cosmetic here. It either silently truncates and degenerates the report, as the comment above these constants already warned, or fails the run, and a failed run is only recoverable via retry, which deletes every plan step, source and document source and nulls the report. Both paths now share _prompt_char_budget plus _trimmable_budget: each trimmable section is measured against what the rest of the prompt leaves, and can reach 0 instead of a floor, because a shorter report beats a destroyed run. Evidence is budgeted before the chat history, since the evidence is the report. Unknown context still keeps the full cap. At 4096 tokens the synthesis prompt now fits (0.6x). Below that it is still over, since a 40-source catalog alone exceeds the window; that needs a smaller maxSources, and the context box does accept values down to 128. test_synthesis_evidence_budget_tracks_loaded_context asserted the old floor at 2048 tokens, which is the bug, so it now asserts 0 and that the rest of the prompt counts against the same budget. Verified: 2325 passed across the research/web/sandbox/chat-history/rag/tool suites. The test_mcp_stdio_sessions failure is pre-existing and fails identically with these changes stashed. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Scope replayed research history to its own attempt for PR #7219 A retry deletes the previous attempt's research_plan_steps, research_sources and research_document_sources rows but keeps its events, and the SSE route attaches one live run snapshot to every event it emits, replayed history included. The step.completed payload carries only position, title, action, input and sourceCount, so that snapshot is the sole source of the excerpt and evidence. On any refresh after a retry, a replayed attempt-0 step was therefore matched against attempt-1's step row by position alone, and start_position resets to 0 after the delete, so the positions line up exactly. The preserved attempt-0 activity then showed attempt-1's excerpt and evidence, or lost them entirely when attempt 1 had not yet reached that position, under a banner that says previous activity is preserved. The run.started resumed branch read the same cross-attempt snapshot and spliced those activities out. Both are gated on the event's attempt matching the snapshot's retryCount, which is the same attempt scoping get_reasoning_text already applies server-side. The excerpt and evidence fall back to what the activity already holds, so a mismatch is non-destructive rather than blanking it. Verified: frontend contract 11 passed, tsc -b exit 0, and the new test fails without the store change. * Retry pre-stream failures in the research stream for PR #7219 _stream_completion serves planning, every decision step and synthesis, and it had no transport retry: a connection error or a 5xx raised before any response byte failed the durable run, and retry then deletes every gathered source, document source and plan step. _completion already treats the identical failures on the identical endpoint as retryable, so the two paths disagreed. This is partly a hole my own 689b06535 opened. After the no-model 400 the body is read, the connection returns to the pool, and _wait_for_local_model then sleeps for up to modelTimeoutSeconds before re-sending on the same client. Uvicorn's keep-alive is 5s, so that pooled connection is essentially always server-closed by then, and losing the has_expired race raises RemoteProtocolError, killing the run the wait existed to save. Also reachable via a read timeout waiting for headers under prompt-eval load. Retrying is safe only because nothing has been consumed at that point, and that is structural rather than a convention: with stream=True httpx returns on the response headers without calling aread(), and raise_for_status() reads no body, both verified against the installed 0.28.1. The handler is scoped to the inner try that ends at break, and _iter_stream_lines sits outside the loop with no path back to send, so a re-send cannot duplicate report text. Bounded and mirrors _completion: same >= 500 predicate, same 3 attempts, same 2**attempt backoff, lease and cancellation re-checked before re-sending. The transport counter and the model-wait counter are independent, so they cannot multiply. The response is closed before every re-send, as manual stream mode requires. Note HTTPStatusError is not a TransportError in httpx, so both are caught explicitly. Verified: 2330 passed. Five of the new tests fail without the fix; the three that pass either way are the invariants that must not change (fail fast on a real 400, never retry once the report has streamed, existing model-wait path). * Bound the planning prompt to the loaded context for PR #7219 Completes dc16598a4, which budgeted the decision and synthesis prompts but left planning unbounded. The question reaches the planner verbatim (a pasted document arrives here as-is) and the history is capped only at the fixed 12,000 chars, so on a small context planning could overflow before any plan was persisted, failing the run without doing any research at all. Same helpers as the other two paths. The question is budgeted before the history, since the question is the request. A test now asserts all three prompt paths hold their own context budget, so a fourth path cannot be added later without one. Verified: 2331 passed; the new test fails without the change. * Keep prompt inputs non-empty and fit the source catalog for PR #7219 Two follow-ups to the prompt budgeting, the first a regression I introduced in dc16598a4. The output reserve was a flat 4096 tokens, so on any context at or below that, including the documented 4096-token GGUF floor, the whole prompt budget came out as 0. Every trimmable section then sliced to nothing: planning_question became the empty string, so the planner never saw the request at all, and synthesis dropped all its evidence. Removing the old floor outright went too far; an empty prompt is worse than the overflow it was avoiding. The reserve is now capped at half the window, and the question and the evidence each keep a floor, since one carries the request and the other carries the answer. A truncated completion is recoverable, a confidently empty report is not. The source catalog was the one section still inserted whole. It holds up to maxSources entries with snippets persisted at up to 4000 chars each, so on a smaller context it alone could exceed the budget while the code responded only by zeroing the evidence and history. It is now fitted first, dropping whole entries from the tail rather than slicing mid-entry, because a half-truncated URL is worse than an absent one: the validator would strip it and the claim would be left uncited. Verified: 2333 passed. All three new tests fail without the change; the question now keeps 1072 chars at a 2048-token context and 4144 at 4096, where both were previously 0. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Tighten Deep Research comments for PR #7219 Post-convergence comment pass over the 40 source files in the PR diff, limited to lines the PR itself adds so untouched upstream code in the same files is left alone. 15 files, 110 insertions, 141 deletions. The reduction is deliberately small. Almost every comment here records why something non-obvious is done, a measured result, a spec rule, or the exact bug it prevents, and those are worth more than the lines they cost, so nearly every edit is a same-meaning compression rather than a deletion. Kept in full: the GFM autolink citation for the URL trim, the catastrophic-backtracking note on _DOCUMENT_CITATION, the prompt-budget notes recording that a reserve at or above the context leaves nothing, the two measured site: filter findings, and the remount note on the activity panel key. Verified comment-only three ways: comment_tools.py reports 15/15 code-unchanged, and an independent ast.dump comparison with docstrings stripped shows zero of the 12 Python files differing. 421 backend tests and the 11 frontend contract tests pass, and the phrase the contract test asserts on is still present on one line. * Harden Deep Research model streams * Fit Deep Research decision prompts * Preserve Deep Research follow-up context * Redact composite credentials from research queries * Scale Deep Research UI typography * Address Deep Research refinement review * Harden Deep Research refinement edge cases * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: danielhanchen <unslothai@gmail.com> Co-authored-by: Daniel Han <danielhanchen@gmail.com>	2026-07-26 23:36:02 -07:00
Souravrajvi0	b448fb5de0	fix(studio): persist connection model selections for remote clients (#7298) ... * fix(studio): persist connection model selections server-side Remote Studio clients could see saved connections but not their enabled model lists because models lived only in browser localStorage. Store models and available_models in llm_providers and sync them through the providers API so alternate clients inherit the same catalog state. Fixes #7281 * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Hydrate external connections on chat startup (#7281) Extract provider sync logic into sync-external-providers.ts and call it from chat-page on mount so persisted model selections appear in the Connected picker without opening Settings → Connections first. * fix(studio): backfill connection models and preserve local options (#7298) Address Codex P2 on remote connection persistence: - Backfill localStorage model selections to /api/providers when backend rows still have empty models_json (legacy upgrades) - Carry promptCacheTtl and openaiContainerTtlMinutes through startup sync - Await hydratePersistedSettings before syncing on ChatPage mount Contract tests: 7 passed; npm run typecheck passed. * Tighten comments * Tighten comments --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Daniel Han <danielhanchen@gmail.com>	2026-07-23 19:11:50 -07:00
Nilay	f3c085ad9e	Fix resume training crash recovery and MLX checkpoints (#6796) ... * Fix resume training crash recovery and MLX checkpoints * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Address review: preserve interrupted stop-and-save output_dir, verify MLX checkpoint - finish_run: add clear_output_dir flag; preserve output_dir for stopped/error unless cancel explicitly clears it (fixes pump finalization wiping persisted path). - training pump: pass interrupted stop-and-save context into finalize_run_in_db. - MLX stop-and-save: verify resumable checkpoint exists before sending complete; return bool from _write_mlx_stop_checkpoint and add regression tests. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Address Codex review: MLX current-step checkpoint and cancel error finalize - Only skip MLX stop checkpoint write when checkpoint-{current_step} exists; stale periodic checkpoints no longer mask missing stop saves. - Pass clear_output_dir through error-event finalization so Stop-without-save cannot leave a persisted output_dir that still offers Resume. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Address review * Address more reviews * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * more reviews * clear in-memory output_dir on interrupted cancel * allow resuming errored runs at the final step * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Clear persisted output_dir in cancel watchdog path * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Write MLX stop checkpoint in stop path, keep output_dir on crash finalize * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix(studio): harden resumable run finalization * fix(studio): defer safetensors checkpoint import * fix(studio): reject stale training cancellation * fix(studio): replay null resume targets * fix(studio): serialize terminal cancellation * Harden resume checkpoint validation and fix stop-save cleanup - Reject unrecognized shard formats and keep indexed shard paths inside the checkpoint dir - Require a non-empty tensor record when validating .pt/.bin optimizer and model state - Always finalize TensorBoard and W&B on stop-save-failure exits - Refuse writing an MLX stop checkpoint through a symlinked directory - Clarify the resume rejection message to cover errored runs * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Tighten resume/checkpoint comments * Recover resumability when a valid stop checkpoint landed - Re-validate the current-step checkpoint in the dead-worker and error finalization paths so a stop-and-save that actually wrote a valid checkpoint is not wrongly marked error/resume_blocked - Accept a valid tensor-free optimizer state (e.g. SGD without momentum); the model-state check still requires real tensors - Include errored runs in the frontend resume rejection message * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Lyxot <longyixing331@gmail.com> Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com> Co-authored-by: danielhanchen <danielhanchen@gmail.com>	2026-07-21 02:34:58 -07:00
Michael Han	65587c2be7	Studio: Data settings tab, uploaded files manager, quant pinning, and chat image preview fix (#7029) ... * Studio: Data settings tab, uploaded files manager, quant pinning, image preview fix Settings - New Data tab in the settings sidebar, under Connections. Chat data management (archived chats, confirm before deleting, exports, import, clear all) moved there from the Chat tab. - New Archive all chats action with confirmation. Archives every chat in Recents and Projects; compare pairs count as one chat. - New Uploaded files manager listing RAG documents (chats, projects, knowledge bases) and chat message attachments with location, size and date. Files can be opened in a new tab or deleted. Deleting a chat attachment keeps the message text. Backend - GET /api/rag/documents lists all uploaded RAG documents with file size plus KB and project names. - GET /api/chat/attachments lists chat message attachments; per attachment file and delete endpoints included. Model selector - Downloaded GGUF quants can be pinned from the quant row (next to the settings and delete actions). Pinned quants show at the top of On Device under a Pinned heading as model name plus a grey quant chip and load directly with one click. Non GGUF cached repos pin as a whole. - Toned down the green of the downloaded label. Fix - Clicking an image attachment in chat now opens the preview overlay. The tooltip trigger wrapper called preventDefault before composed handlers ran, which made Radix DialogTrigger skip opening. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: image previews and file type chips in uploaded files list Image attachments now show a small thumbnail (lazy loaded from the stored bytes, object URL revoked on unmount) and every row shows a grey uppercase type chip derived from the extension or content type. Non image rows keep a file icon. Name cell floors its width and clips overflow so narrow dialogs stay aligned. * Harden attachment serving, add tests, and polish pinned rows and previews - Strict base64 decoding for attachment files: corrupt payloads now return 422 instead of silently serving empty or garbled bytes; whitespace, missing padding, the URL-safe alphabet, and RFC 2397 percent-encoded data URLs are all handled - New backend test suite covering attachment listing, size accounting, malformed rows, deletion semantics, and every file-serving edge case - Pinned quant rows show a Loaded tag when that exact quant is active, and reveal unpin, settings, and delete actions on hover - Uploaded files dialog is wider and chat locations link straight to the thread the attachment belongs to - Chat image preview is now a chrome-free lightbox: dimmed backdrop, rounded image, corner close button, click outside to dismiss - File opens go through a synchronous window.open so Safari and Firefox popup blockers do not eat them * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Uploaded files: click a file to jump to its chat, square thumbs, new Data icon - Clicking a file row (thumbnail or name) now goes straight to the chat it belongs to; files without a chat open directly as before - File thumbnails pin a small 7px radius: the theme scales rounded-md up to a near circle at this size - Settings Data tab now uses the database-setting icon * Uploaded files is now a Data tab subpage instead of a popup - Manage swaps the tab body for an inline Uploaded files page with a back header, matching the rest of settings navigation - Size column header and values are left aligned like the other columns - Column widths tightened so the table fits the settings panel * Lightbox polish and Data tab row order - Image preview close button is transparent until hovered - Preview image no longer rounds its corners - Import chats now sits below Clear all chats in the Data tab * Data tab: export chats as fine-tuning data and open them in Recipes - New Fine-tuning section in Settings > Data converts every chat into a JSONL dataset in the OpenAI messages format, one conversation per line with string-only system/user/assistant turns - The Train tab detects this file as chatml natively: no column mapping and no standardization pass, and it works with train on completions since every assistant turn sits behind the chat template response marker - Consecutive same-role turns merge, trailing turns without an assistant reply drop, and reasoning, tool calls, and images are excluded so chat templates format the data cleanly - Open in Recipes stages the JSONL as a local seed upload, creates a new Data Recipe with the seed block preconfigured, and jumps to the editor * Data tab: load chats straight into the Train tab, row moved to the top - New Load in Train tab button uploads the fine-tuning JSONL through the training dataset endpoint, selects it in the training config store, and opens the Train tab with the dataset loaded and format-checked - Use chats as training data now sits at the very top of the Data tab - The Chats subheading is gone; chat rows flow directly under it * Address review findings on the uploads manager and quant pins - Deleting the last attachment stores '[]' instead of NULL: a NULL reads back as a missing field and triggers the legacy IndexedDB backfill, which resurrected the deleted attachment on the next chat load - The attachment file endpoint now serves audio: adapter parts store {data, format} raw base64 and compare chats store a bare base64 string; media type comes from the attachment contentType or the format - Compare-chat uploads live in message content parts, not attachments; the uploads list now includes those blobs via synthetic content-part ids that the same get and delete routes resolve - Deleting a quant from the expanded repo row also unpins it so a pinned row cannot try to load a file that no longer exists - Thumbnails in the uploads list fetch their blob only once the row is visible, so a long screenshot history does not download everything - Nine new backend tests cover audio serving, content-part listing, serving, deletion, and the empty-list delete behavior * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Data tab: single action dropdown with format choices for chat training data - The three fine-tune buttons collapse into one dropdown plus a run button; pick Load in Train tab, Open in Recipes, or Export JSONL, then click the arrow to run it - The dropdown's Format section adds ShareGPT and Alpaca alongside the default OpenAI messages format, ticked like a checklist; all three shapes are auto-detected by the Train tab's format check - Alpaca is single-turn, so each user to assistant pair becomes its own record with the system prompt and earlier turns carried in the input column - Shorter description on the training data row - Uploaded files rows show the size under the file name instead of a separate column, matching the tighter layout * Polish the training data action control - Run button is a true circle (icon-sm plus rounded-full) with a heavier arrow stroke - Dropdown trigger uses the shared standard chevron and a fixed width so switching actions no longer resizes the control * Shorten the training data row description * Use the standard chevron for the run button and enlarge the ticks - Run button uses the shared standard right chevron so it matches the dropdown chevron instead of the hugeicons arrow - Dropdown ticks bumped up a size for legibility * Reword the training data row description * Shorten Data Recipes to Recipes in the training data description * List Export JSONL first and rename the default format to Chat Completions * Handle legacy string content in fine-tune exports and gate Train on chat-only hosts - messageToPlainText now accepts plain-string message content, the shape legacy and imported histories store, so those conversations export instead of being skipped as having no exchange - The Load in Train tab action is disabled on chat-only hosts the same way the sidebar gates Train; the default action falls back to Export JSONL there so the run button never uploads a dataset that /studio would immediately redirect away from * Narrow the training data action dropdown slightly * Drop the format picker from the training data dropdown Chat Completions (OpenAI messages) is the only export format we ship, so the ShareGPT and Alpaca options and the Format section are removed. The export always uses the OpenAI messages shape. * Address the second round of review findings Security - Chat attachment data URLs no longer echo their embedded media type: anything that is not a plain raster image serves as octet-stream, so imported text/html or SVG payloads cannot render under the app origin - Uploaded .html/.htm RAG documents serve as text/plain for the same reason; the preview sheet only uses the file URL for PDFs Uploads manager - Remote image URLs in imported chats are no longer listed as stored uploads (nothing to serve, and delete would strip the chat reference); the delete guard mirrors the same data:-only rule - Deleting a content-part upload refetches the list since the remaining parts re-index, keeping sibling row ids current - Deleting a project document from the Data tab invalidates the project sources cache like the sources panel does - Data-tab deletions now patch the loaded thread's in-memory copy via a small event, so a later repo sync cannot write the attachment back Fine-tune export - Branch siblings from retries stay out of the exported conversation; only the selected chain converts (full exports still keep everything) - Assistant turns before the first user turn drop, preserving leading system prompts, so no unconditioned assistant targets are emitted Four new backend tests cover the media type clamp and remote-URL rows; two existing tests updated for the clamped types * Fix uploaded file lifecycle and model state * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Make archived chats a Data settings subpage * Studio: fix attachment route tests and pinned quant edge cases - test_chat_attachments: drop asyncio.run around the synchronous /attachments routes (list/get/delete are plain def, so asyncio.run raised 'a coroutine was expected' and failed the Repo tests CI job). - test_chat_attachments: align compare-chat content-part assertions with the stable content-hash id scheme (content-part-sha256-...) instead of the removed array-index ids; resolve ids from the listing. - pickers: pass disabled={deleteDisabled} to the pinned-quant delete action so a quant cannot be deleted mid model-load, matching the expanded variant rows. - pickers: build the pinned-quant existence set from the query-unfiltered cached GGUF repos (format filter still applied) so a pinned quant stays findable when the search term matches only its quant name. * Fix Studio review regressions * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Guard fine-tune export content blocks * Add Export button for archived chats Adds an Export action to the Archived chats view in Settings > Data that downloads only the archived chats as a JSON backup (their threads, messages and projects). The button sits in the archived header row and appears only when archived chats exist. * Refactor archived export into pure, testable units Split the archived-chats export into a dependency-free filter (archived-chat-export.ts) and a shared JSON download helper (download-json.ts). Skip the download when nothing is archived so a stray call never drops an empty file. No behavior change to the button. --------- Co-authored-by: shimmyshimmer <shimmyshimmer@users.noreply.github.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Etherll <61019402+Etherll@users.noreply.github.com> Co-authored-by: Unsloth <michaelhan@Michaels-MacBook-Pro.local> Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>	2026-07-20 04:57:44 -07:00
Gaurav Dubey	dc65638b7d	Studio: expose Windows drive roots in the folder browser (#7082) ... * Studio: expose Windows drive roots in the folder browser The model-selection folder browser bounds navigation to the roots returned by _build_browse_allowlist(), which exposed Linux removable-media mounts via linux_run_media_mount_roots() but had no Windows analog. As a result a user on C: could not browse to D:/E: to pick a model directory. Add windows_drive_roots(), a Windows-only companion to linux_run_media_mount_roots() that lists readable logical drive roots, and wire it into both browse-allowlist builders and their suggestion chips so other drives are both navigable and offered as quick-picks. The helper is a no-op on Linux/macOS, so existing platforms are unaffected. Closes #6368 * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: cover the Windows drive-root browse wiring with an integration test Add an allowlist integration test mirroring the Linux side's test_legacy_browse_allowlist_includes_linux_run_media_mounts: it extracts _build_browse_allowlist from routes/models.py, stubs external_media so windows_drive_roots() yields a fake drive root, and asserts that root becomes browsable through the built allowlist. Proves the wiring, not just the helper. * Studio: skip inactive drives via GetLogicalDrives before probing Resolve active logical drives from GetLogicalDrives() before probing each letter with os.path.isdir. Probing a drive letter mapped to a disconnected network share can otherwise block the async backend for tens of seconds per letter. The call degrades gracefully (falls back to probing all letters) when ctypes/windll is unavailable, so behavior is unchanged on Linux/macOS. Tests override the bitmask source to stay deterministic on real Windows hosts. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: allow browsing descendants of a drive-root allowlist entry routes/models.py _is_path_inside_allowlist() checked descendants with startswith(root_real + os.sep). A drive root ("D:\") already ends in a separator, so the prefix became "D:\\" and a child like "D:\models" was rejected with 403 after the browser opened the drive root. Only append a separator when the root does not already end in one. folder_browser.py already uses commonpath and was unaffected. Adds a regression test covering the separator-terminated-root descendant case. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: enforce the system-directory denylist during folder browsing Exposing whole Windows drive roots (and any legacy-registered filesystem root) widened the browse allowlist above system directories, but the browse resolvers only re-applied the credential/config denylist, not the _denied_path_prefixes() system-dir denylist that scan-folder registration enforces. That let browse-folders enumerate C:\Windows, C:\Program Files, /etc and /proc. - Add is_denied_system_path() to both storage modules and enforce it in both browse resolvers (legacy routes/models.py and hub folder_browser.py), on each resolved child and on the final target, keeping the /run/media carve-out. - Rework the legacy _is_path_inside_allowlist to use splitdrive + commonpath so a Windows drive root authorizes its descendants while a bare POSIX / does not, and to compare case-insensitively like the hub browser. - Reject the filesystem root in the legacy add_scan_folder, matching the hub. - Hide denied system dirs from browse listings and suggestion chips. - Add tests/test_browse_denylist.py and update the external-media path tests. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: make browse-denylist tests OS-portable The browse-time denylist tests used real /etc and tmp_path locations; on macOS tmp lives under the (legitimately denied) /private/var and /etc resolves to /private/etc, so three tests failed there. Pin the platform / use a tmp-based denied prefix so they assert the same behavior on Linux, macOS and Windows. * Studio: apply the bare POSIX-root guard to the hub folder browser too The _is_path_inside_allowlist guard that stops a legacy-registered '/' scan folder from authorizing every absolute path lived only in the legacy browser. The hub browser used commonpath without it, so a stale '/' row let it descend into /var, /root, /home -- which the system-directory denylist (/proc /sys /dev /etc /boot /run) does not cover, while the legacy browser blocked them. Mirror the legacy guard so both browsers treat '/' identically. Also resolve each directory entry before the denylist check in both listing loops, so a symlink or junction pointing into a denied dir is hidden instead of rendered as a row that 403s on descent. Adds legacy-vs-hub parity tests. * Studio: bound Windows drive probing so a disconnected mapping can't stall the browser GetLogicalDrives includes mapped network drives, so a disconnected but still mapped drive (e.g. Z: -> \\nas\share) stays set in the bitmask and reaches os.path.isdir, which can block for tens of seconds while Windows tries to reconnect. Because windows_drive_roots() runs synchronously while building both folder-browser responses, one stale mapping stalled every browse request. Probe each surviving drive in a daemon thread bounded by a short timeout and skip it if it does not answer in time, so a hung mapping is dropped instead of blocking the caller. Connected drives (local or network) still respond well within the timeout, so drive discovery is unchanged. Corrects the GetLogicalDrives docstring, which claimed the bitmask alone prevented the stall. * Studio: probe drive/media roots once per browse request, not twice Both folder browsers called windows_drive_roots() (and linux_run_media_mount_roots()) twice per browse request: once to seed the allowlist in _build_browse_allowlist() and again to build the suggestion chips. With the bounded drive probe, a disconnected mapped network drive then paid the timeout twice per folder click. Probe both once in the request handler and pass the results into _build_browse_allowlist(), reusing them for the chips, in both the legacy and hub browsers. Adds a test asserting the roots are reused, not re-probed. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: run the legacy browse endpoint in the threadpool, fix its stale test Two follow-ups from review of the drive-probe changes: - browse_folders was 'async def' but does only blocking filesystem I/O (the timeout-bounded drive probe, iterdir, realpath). On the event loop a disconnected mapped drive waiting out its probe timeout stalled every other request. Declare it sync 'def' so FastAPI runs it in the threadpool, matching the hub browse endpoint. No await was used in the body. - test_browse_folders_hides_sensitive_dirs monkeypatched _build_browse_allowlist with a zero-arg lambda; the once-per-request refactor now calls it with (media_roots, drive_roots), so the lambda raised TypeError. Accept and ignore the args. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: probe Windows drive roots concurrently so multiple dead mappings don't stack timeouts windows_drive_roots() probed each candidate serially, so N disconnected-but-mapped network drives each paid the full per-drive timeout in turn (e.g. four stale mappings added ~8s to every folder-browser request). Collect the candidate roots first, then probe them all at once under a single overall deadline, so the added delay stays at ~one timeout regardless of how many drives are disconnected. _readable_dir_within stays as a thin single-path wrapper for its existing callers/tests. * Studio: tighten comments in the folder-browser drive-root changes Condense the comments and docstrings added by the Windows drive-root and system-directory denylist work to be shorter and clearer while keeping the security and correctness rationale intact. Comment and docstring text only; no code changes. * Studio: iterate the input, not the results dict, when collecting readable drive probes _readable_dirs_within returned {path for path, ok in results.items()...}, but a probe thread that exceeded the join deadline is still alive and can insert its key into results during that iteration, raising 'dictionary changed size during iteration' -- reachable exactly in the disconnected-mapped-drive case the probe exists for. Iterate the fixed input list and read results.get(path) (an atomic read) instead. * Studio: keep the browse-route containment tests denylist-inert so they pass on macOS test_browse_folders_route.py exercises allowlist containment and the file-vs-directory guard, not the system-directory denylist. On macOS pytest tmp_path resolves under /private/var, a denied prefix, so _resolve_browse_target 403s the fixture dirs before the containment logic runs (4 failures). Add an autouse fixture that makes is_denied_system_path inert in this file; the denylist keeps its own coverage in test_browse_denylist.py. * Studio: keep the hub browse tests denylist-inert so they pass on macOS * Studio: register a UNC share root; only reject local filesystem roots * Studio: reject device drive roots and browse a registered UNC share root * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: treat device-namespace volume GUID roots as local filesystem roots --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: danielhanchen <danielhanchen@gmail.com>	2026-07-15 00:24:11 -07:00
Nilay	07ecdb34c0	Sort chat recents by last activity (#6844) ... * show chat by by last activity * Update chat thread updated_at logic and enhance sidebar chat item handling * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-07-07 17:54:32 +01:00
ramisworld	01f7e14988	Fix Studio custom folders on Linux external drives (#6799) ... * Fix external drive custom folder selection * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update studio/backend/tests/test_linux_external_media_paths.py Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Keep legacy media scan validation strict * Apply sensitive-dir denylist to legacy folder browser for PR #6799 The legacy /api/models browse endpoint gained the new /run/media mount roots in its allowlist but not the credential/config guard that scan-folder registration and the Hub browser already enforce. Filter sensitive names during enumeration and reject them in _resolve_browse_target so .ssh, .aws, .config, etc. under allowlisted roots stay unbrowseable, matching the Hub browser. Add a public contains_sensitive_path_component helper and cover the legacy resolver with a regression test. * Trim redundant comments in PR #6799 changes * Skip sensitive Linux media roots * Reject sensitive dirs at exact browse roots for PR #6799 Both _resolve_browse_target functions only checked contains_sensitive_path_component while walking descendant parts, so requesting an allowlisted root itself (empty relative path) returned it unchecked. A pre-existing scan-folder row under ~/.ssh, ~/.aws, ~/.config, etc. (registerable before the denylist was added) is re-added to the allowlist on upgrade and could then be browsed. Check the resolved target once before returning in both the legacy and Hub browsers, and cover the root case in both test suites. * fix: avoid unused path helper reexports * fix: import sensitive path helpers directly * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> Co-authored-by: Daniel Han <danielhanchen@gmail.com> Co-authored-by: imagineer99 <samleejackson0@gmail.com> Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>	2026-07-03 19:10:04 +01:00
Michael Han	4f24b12cc9	Studio: customizable RAG embedding model with HF search, settings tab reorganization (#6800) ... * Add customizable RAG embedding model setting and reorganize settings tabs Chat with files, project sources, and knowledge bases previously always embedded with unsloth/bge-small-en-v1.5. This adds a Settings option to pick any Hugging Face embedding model (or local path), with HF search autocomplete, server-side verification that the repo is actually an embedding model, and a save anyway escape hatch for offline or local models. The setting persists in app_settings and applies at runtime to both the sentence-transformers and llama-server GGUF embedder backends without a restart. Also reorganizes the General settings tab: Documents & RAG sits above Uploads, Helper LLM moved above the danger zone, and Model auto-switch (OpenAI API) moved to the bottom of the API tab. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Support local model paths on the GGUF embedder and normalize default saves Found by simulation testing of the embedding model setting: Local paths saved as the embedding model now work on the llama-server GGUF backend (the default backend on macOS and CPU). A path to a .gguf file is used directly and a directory is scanned for a variant-matching non-mmproj .gguf, with a clear error when none exists. Previously a local path was sent to the HF hub API and failed with a repo lookup error. Saving the default model explicitly no longer stores an override, so is_custom stays false and the UI does not show a reset button for the default value. * Address review: stale-vector handling, GGUF derivation, save-time guards Review follow-ups, each verified by new tests: Re-uploading a document after an embedding model change now re-indexes instead of deduping by content hash. Documents record the embedder that produced their vectors (lazy embedding_model column, NULL legacy rows keep deduping) and a mismatch replaces the old document. A vector width change no longer bricks the dense index. ensure_vec drops and recreates chunks_vec when the dim changes (old vectors are in a foreign space and only block inserts) and search_dense returns empty on a width mismatch instead of surfacing a vec0 error, so lexical search keeps working until documents are re-uploaded. Saving a local sentence-transformers folder with no .gguf now returns 409 with a clear message when the install embeds via llama-server, instead of failing at first index. force still saves. A custom RAG_EMBEDDING_MODEL env without RAG_EMBED_GGUF_REPO now derives the -GGUF companion repo instead of silently keeping the bge GGUF on CPU and macOS installs. The resolved GGUF path is tagged with the repo captured at entry, so a setting change during a download cannot mark the old model as current. GGUF repo detection matches gguf as a whole name segment rather than a substring, hf_token is trimmed before verification, and the settings combobox drops a redundant state mirror of its controlled value. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Shrink embedding model font to 11px in the input and dropdown The combobox wrapper applies className to the outer input group, so the size utility must target the inner input element; the previous text-xs never reached it and the field rendered at the browser default. * Show curated unsloth embedding models when the search field is empty The empty-query listing was the global top-downloads page, which holds no unsloth mirrors for the unsloth-first float to reorder, so the dropdown opened on third-party models. Match the model picker: curated unsloth listing when empty, whole-Hub search once a query is typed. * Address review: settings resilience and index consistency Keep the last known embedding model on settings store errors, remove the re-entrant dim lock in the llama-server backend, accept local GGUF saves and verify GGUF availability for HF repos on that backend, match local path embedders exactly in model list filters, drop same-width stale vectors from dense search, pin the embedder per ingestion job, and only replace completed documents after the re-index succeeds. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Consolidate the GGUF repo derivation tests * Trim to a single core embedding-model test * Address review: GGUF repo saves and cache race Accept a GGUF-named HF repo on the llama-server backend by verifying GGUF availability instead of the sentence-transformers metadata gate, and guard the settings cache with a generation counter so a read overlapping a save cannot repopulate it with the pre-save value. --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-07-02 05:26:33 -07:00
Daniel Han	5211b506e1	Studio: opt-in OpenAI /v1 model auto-switch and idle keep-warm (#6392) ... * Studio: opt-in OpenAI /v1 model auto-switch and idle keep-warm The OpenAI-compatible endpoints serve whichever GGUF is loaded and ignore the request model field, so an OpenAI client that changes model never reloads. Add an opt-in setting that, when a /v1 request names a downloaded local GGUF different from the loaded one, loads it before serving by reusing the existing /load path (its dedup, tensor fallback, and threading apply). Unknown names still serve the loaded model, so drop-in compatibility is preserved and no remote download is triggered. Also add an optional idle auto-unload (TTL keep-warm): a pure-ASGI middleware tracks in-flight inference requests so a stream is never unloaded mid-response, and a lifespan loop unloads the model after the configured idle seconds. Both settings default off and live in the app_settings store, exposed via GET/PUT /api/settings/openai-auto-switch. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: variant-aware auto-switch, /v1/responses coverage, keep-warm load stamp Follow-ups from review of the opt-in OpenAI auto-switch path: 1. Variant-aware dedup. _maybe_auto_switch_model compared only the repo id, so requesting another quant of the loaded repo (e.g. Q4_K_M loaded, Q8_0 asked) was served by the old quant. Compare hf_variant too, matching /load dedup. 2. Streaming /v1/responses now calls the auto-switch hook. It went straight into _responses_stream and only checked is_loaded, so stream=True could serve the old model or 400. Non-streaming already routed through chat completions; the hook is idempotent once loaded. 3. resolve_local_gguf tries an exact id match before splitting a trailing :VARIANT, so local ids that contain a colon (e.g. a Windows path) resolve instead of being cut at the drive letter. 4. Idle keep-warm stamps activity on a load/swap transition. _last_active was only refreshed by inference requests, so a model loaded after the server sat idle past the TTL could be unloaded before its first request. Tests cover each case. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: make the /v1/responses auto-switch test order-independent The new streaming-responses test passed in isolation but failed under the CI's randomized collection order with "object has no attribute 'state'": it passed a bare object() as the request and stubbed only one dispatcher, so an ordering where the real dispatcher ran hit request.state. Give the request a state and stub both dispatchers; the test still asserts the hook fires before dispatch. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: assert /v1/responses auto-switch wiring on source, not at runtime The behavioral version executed openai_responses and relied on stubbing its callees, which a randomized collection order in CI could defeat (the real dispatcher ran and hit request attributes). Assert on the function source that the hook precedes both dispatchers instead; the hook's runtime behavior is already covered by the direct _maybe_auto_switch_model tests. * Studio: auto-switch on /v1/embeddings, GGUF-only targets, idle-unload race gate Second-pass review follow-ups on the opt-in auto-switch path: 1. /v1/embeddings now calls the auto-switch hook before the loaded-state check, matching the other model-bearing OpenAI endpoints (the keep-warm middleware already treats embeddings as inference). 2. The resolver index is now GGUF-only. The local-model scanners also surface Transformers/safetensors repos; without a filter, auto-switch could unload the GGUF and route a request into the non-GGUF loader. _has_local_gguf checks a direct .gguf, a models-dir folder, and the HF-cache snapshots layout. 3. Idle keep-warm now holds an asyncio gate across the idle check and the unload, and a request bumps inflight under the same gate, so the loop can no longer unload in the window between "looks idle" and the kill. Tests cover each. Broader local-model source parity (LM Studio, Ollama, legacy caches, custom scan folders) is a follow-up; missing one of those today just falls through to the loaded model. * Studio: variant-aware local resolver, count_tokens + audio auto-switch coverage Third-pass review follow-ups on the opt-in auto-switch path: 1. The resolver is now variant-aware via list_local_gguf_variants. It indexes only the quants actually on disk, recursing snapshots and quant subdirs such as the nested per-quant folders, so a requested repo:VARIANT resolves only when that quant is local and a bare repo resolves to a concrete local quant. This fixes two gaps: the previous shallow glob rejected nested-variant GGUF repos, and a request for an uncached quant could send /load down the remote download path, breaking the local-only contract. 2. /v1/messages/count_tokens now auto-switches like its sibling /v1/messages, so a count uses the requested model's tokenizer. 3. /api/inference/audio/generate (direct GGUF TTS) is now tracked as in-flight inference, so the idle loop cannot unload the model mid-generation. Tests cover each. Two reviewer items are left as follow-ups: indexing the remaining local sources (LM Studio, Ollama, legacy/default caches, custom scan folders), which fails safe today by falling through to the loaded model; and fully serializing concurrent different-model requests, an inherent limit of the single-slot llama backend that the opt-in feature is not designed around. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: make local GGUF resolver fail-safe so a bad model name cannot 500 The auto-switch hook calls resolve_local_gguf without its own guard, and /v1/completions and /v1/embeddings pass body.get("model") through unchanged. A non-string model (e.g. {"model": 123}) or any internal scan failure would then raise out of the resolver and turn a request that would otherwise be served by the loaded model into a 500, breaking the drop-in compatibility the feature is built on. Guard the resolver at its boundary: reject non-string input up front and wrap the lookup so any failure returns None (fall through to the loaded model). Add regression tests for both paths. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: per-model launch flags for auto-switched GGUF models * Studio: list switch-eligible GGUFs in /v1/models when auto-switch is on * Studio: settings UI for OpenAI model auto-switch and idle auto-unload * Studio: show save error over the disabled-idle hint in auto-switch settings * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: address gemini review (case-insensitive /v1/models retrieve, idle-input empty guard) * Studio: address codex review (deterministic override args, exclude probe/embedding models from discovery) * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: keep-warm count_tokens, gate idle on auto-switch, drop hidden models Three hardening fixes to the opt-in auto-switch path surfaced while reviewing the work that builds on it: 1. count_tokens keep-warm. /v1/messages/count_tokens counts via the loaded tokenizer and already auto-switches, but the keep-warm middleware did not track it, so idle auto-unload could free the model mid-count. It is now a tracked in-flight path. 2. "Off means unchanged" for idle unload. get_auto_unload_idle_seconds now reports 0 while auto-switch is disabled. Idle unload only makes sense with auto-switch on (an unloaded model returns only via the next request's swap), so a stray TTL can no longer trigger a destructive unload while the feature is off, keeping the disabled state identical to pre-feature behavior. 3. Hidden models are not switch targets. The resolver index now skips what Studio hides from its own pickers (the llama.cpp validation probe, RAG embedding weights) via _is_hidden_model, so they can never be auto-switched to by name. Tests added for each. * Studio: bare-id reuse, responses validation order, in-flight tracking Review follow-ups after folding in the per-model overrides and discovery work: 1. A bare model id (no :VARIANT) is now satisfied by any loaded quant of that repo. Previously a bare name resolved to the largest local quant, so it could force a slow reload when a different quant of the same repo was already serving. An explicit repo:VARIANT request still honors the quant. 2. /v1/responses now runs the auto-switch hook after the empty-input validation so a request that 400s can no longer trigger a multi-minute model load before being rejected. The hook still precedes both dispatchers, so streaming requests switch. 3. The keep-warm middleware now tracks in-flight requests whenever auto-switch is enabled rather than only when the idle TTL is already positive, so a stream that starts with the TTL at 0 is still protected if idle-unload is enabled mid-stream. Off still passes straight through. Tests added for each. * Studio: tighten auto-switch code comments Comment/docstring-only pass over the OpenAI auto-switch feature: collapse multi-line blocks, drop a comment that restated the gate it sits next to, and trim verbose docstrings on internal helpers while keeping the load-bearing rationale (concurrency, API behavior, drop-in compat, gotchas). No logic change: verified comment-only with the AST/printer signature check. * Studio: bind auto-switch locks per running loop Review follow-up. The auto-switch swap lock and the keep-warm unload gate were module-level asyncio.Lock objects. That is safe under the single uvicorn loop and on Python 3.10+ (the Lock resolves the running loop lazily on acquire), but a module-level Lock binds to one loop on pre-3.10, which can raise a loop mismatch in multi-loop runners. Resolve each lock through a per-loop accessor backed by a WeakKeyDictionary so every running loop gets its own Lock and stale loops are collected. No behavior change under the server's single loop. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: auto-switch re-review fixes (body codes, coverage, swap, alias, tracking) Follow-ups from a second review pass over the opt-in OpenAI auto-switch feature: 1. OFF-state status codes: /v1/completions and /v1/embeddings moved the body read ahead of the loaded-state check, so a malformed/empty body with no model loaded returned 500 instead of the prior 503. A shared helper reads the body defensively (an unparseable/non-dict body yields no model), and the handler re-reads after the 503 gate to surface the original parse error exactly as before. OFF behavior is unchanged. 2. Local-model coverage: the resolver index only scanned ./models and the active HF cache, while the model picker also lists the legacy/default HF caches, LM Studio dirs, and user scan folders. A request for one of those named models silently served the loaded model instead. _build_index now scans the same roots (Ollama's symlink-creating scanner is skipped on the request path), and resolution is offloaded with asyncio.to_thread so the wider scan never blocks the event loop. 3. Swap vs in-flight stream: a cross-model swap killed the llama-server while another client was still streaming from it. The hook now tracks how many requests are streaming on the loaded model (in-flight minus those still inside the hook) and returns 409 instead of swapping while one is active. Concurrent same-model requests never reach this path, so they are unaffected. 4. Idle-unload + alias: after idle-unload freed the model, an unknown/alias name resolved to nothing and 503'd, though it served the active model before the TTL. Idle-unload now remembers the freed id and an alias request reloads it (only an already-local model, so no remote download), cleared once a model is loaded again. 5. In-flight tracking: the keep-warm middleware tracked in-flight only while the feature was on, so a stream started while off could be unloaded if idle-unload was enabled mid-stream. It now tracks on every inference path; counting is cheap and invisible to clients. Tests added for each. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Remove stray async_task_outputs files committed by mistake * Studio: auto-switch review round 3 (revert swap guard, hardening) Addressing a third review pass: - Revert the cross-model swap guard. It counted keep-warm in-flight (which includes external-provider calls that never touch the local model) and so could 409 a local swap spuriously, and it still left a same-model request able to start streaming on the model a concurrent swap was unloading. A correct fix needs a request-lifetime reader/writer barrier; a partial guard was worse than the honest single-slot behavior, so concurrent different-model use is back to being serialized (documented), like llama-swap's single slot. - Non-string request model (e.g. {"model": 123} on a raw-body endpoint) is now treated as absent, so it falls through instead of raising in the membership checks once an idle-unload stash exists. - Idle-unload now stashes and replays the freed quant: an alias reload restores the exact (id, variant) that was freed rather than the largest local quant. - Anthropic /v1/messages validates max_tokens before the auto-switch hook, so a request that 400s never triggers a model load. - Keep-warm tracks a pending count for requests waiting on the unload gate, so the idle loop cannot unload the model out from under a request that is blocked on the gate but not yet counted as in-flight. - The idle-unload task is awaited after cancel on shutdown to avoid pending-task warnings. - The resolver's HF cache scan is None-safe and logs at debug instead of letting a bad root abort the whole index build. - upsert_app_setting_map_entry rolls back explicitly on error. Tests updated/added for each. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: keep saved idle-unload seconds when auto-switch is toggled off * Studio: auto-switch hardening (thread-safe lock maps, body validation) Defensive fixes from review: - Guard the per-loop WeakKeyDictionary get-or-create for both the unload gate and the auto-switch lock with a threading lock, since WeakKeyDictionary mutation is not thread-safe when two event loops run on different threads. - Build the resolver index under the cache lock so concurrent callers with an expired cache don't all run the multi-dir scan at once. - /v1/completions and /v1/embeddings return a clean 400 for a valid JSON body that is not an object (e.g. a list), instead of a 500 from body.get(...). - The keep-warm middleware only tracks POST requests (inference is always POST), so CORS preflight (OPTIONS) is not counted, and tolerates a None path. Tests added for the list-body 400 and the non-POST skip. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: auto-switch review round 4 (local-path load, swap guard, idle fixes) From a 10-reviewer pass: - HF-cache entries now load by a concrete local path, not the bare repo id. The resolver records a load_path (the snapshot dir for a models--* cache repo, the file/dir otherwise) so /load takes the local branch and can never trigger a download to satisfy a partial cache. The advertised loader_id (repo id) is kept as the launch-override key. resolve_local_gguf now returns (load_path, variant, loader_id). - Re-add a single-slot swap guard: a cross-model swap returns 409 model_switch_busy while another inference request is active rather than killing its stream (the caller is excluded from the count), and holds the keep-warm gate across the load so no new inference starts mid-swap. Concurrent same-model requests never reach this path. A residual spurious 409 is possible while a concurrent or external- provider request is active; that is the documented single-slot tradeoff. - Idle keep-warm tracks (model_identifier, hf_variant): reloading the same repo at a different quant counts as a fresh model, so it is not unloaded before one TTL. - Track Studio's own /api/inference/generate/stream so the idle loop can't unload the model mid-stream on that route. - A successful manual /load clears the idle-unload reload stash synchronously, not only on the next idle poll. Also merged origin/main (the branch had fallen behind, which would have reverted unrelated files on merge). Tests added/updated for each. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: auto-switch review round 5 (concurrency, identity, load gate) From a 10-reviewer pass (9 request-changes, 1 approve): - Concurrent same-target requests load once instead of each returning 409. The count-based busy guard could not tell "another request wants the same model" (safe, load once) from "another request is using the loaded model" (refuse). Track in-flight auto-switch requests per (target, variant) and subtract same-target waiters from the busy count; a cross-model swap still 409s while a genuinely different request is active. - Fix the identity confusion introduced when round 4 began loading by concrete local path: the backend identifier became a filesystem path. Record the advertised repo id on the backend after an auto-switch load and use it so (a) a model loaded manually by repo id is recognized as already serving (no spurious reswap/409), (b) /v1/models reports the repo id, never a host path or a duplicate, and (c) the idle-unload stash keeps the override keyed by the repo id, so an alias reload after TTL keeps the user's saved launch flags. - Gate the manual /load route with the keep-warm lifecycle gate so idle auto-unload can't unload a model mid-load. load_model now wraps _load_model_impl in the gate; auto-switch calls _load_model_impl directly since it already holds the gate. - Restore default-off parity on Anthropic /v1/messages: an unloaded backend with auto-switch disabled 503s before the max_tokens 400 check, as it did pre-feature. When the feature is on, request-shape validation still runs before any load. Tests added for each; full backend suite diff vs baseline is unchanged. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: auto-switch review round 6 (concurrency ordering, leaks, unload gate) From a second 10-reviewer pass (8 request-changes, 2 approve): - Same-target concurrency: register a waiter by the raw requested model before the (slow) resolve, and exclude pending requests from the swap busy count. The middleware counts a concurrent same-model request as in-flight before it resolves and joins the resolved-target waiter map, so the prior fix could still 409 it. The guard now subtracts max(same resolved-target, same raw-request) waiters and ignores pending (a pending request is blocked in the middleware, not generating, so a swap can't interrupt it). - External-provider requests no longer block a local swap. The keep-warm middleware counts every inference-path POST, but external-provider chat returns before the auto-switch hook and never touches the local GGUF. The chat handler now untracks itself before proxying, so its in-flight stream can't trip model_switch_busy on a concurrent local auto-switch. The middleware skips its own end-decrement for an untracked request. - Manual /unload is gated like load and idle-unload: it holds the lifecycle gate and returns 409 rather than tearing down llama-server while an inference request is in flight. - Response model id no longer leaks the load path. /v1/models already advertised the repo id; chat, completions, embeddings, Anthropic messages, and audio response bodies now use the same _llama_public_model_id helper instead of the concrete on-disk model_identifier. - Chat completions validates the non-system-message requirement before the auto-switch hook (as /responses and /messages already do), so an invalid request can't swap the resident model before returning 400. Tests added for each; full backend suite diff vs baseline is unchanged. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: auto-switch review round 7 (teardown policy, Unsloth-active swap, training) From a third 10-reviewer pass (9 request-changes, 1 approve), all on the same asymmetric-teardown theme. Resolved per the intended policy that only automatic paths defer to an active stream; deliberate user actions stay interrupting: - Revert the manual /unload in-flight guard added last round. A manual /load or /unload is a deliberate action and tears down immediately, as before; only the automatic idle-unload loop and auto-switch defer to an active request. This removes the asymmetry the reviewers flagged (manual /load, the /unload Unsloth branch, and the opposite-backend swaps inside _load_model_impl) by not extending the guard to deliberate paths, rather than spreading it. - Auto-switch now refuses a swap whenever another inference request is in flight, not only when a GGUF is already loaded. _load_model_impl also unloads an active Unsloth/transformers backend before loading a GGUF, so the busy guard must cover that case too; otherwise an Unsloth stream could be killed by an auto-switch. - Refuse API-initiated training while inference is active. When Studio is driven as an inference API (sk-unsloth key auth), POST /api/training/start returns 409 if a request is in flight, since training frees VRAM by unloading the chat model and would kill the stream. The Studio UI (session auth) still starts training and coexists/frees VRAM as before. A mixed UI+API session is not yet special-cased. Adds auth.authentication.authenticated_via_api_key. Tests added/updated for each; full backend suite diff vs baseline is unchanged. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: add UNSLOTH_MODEL_IDLE_TTL env override for idle-unload Borrowed from PR 6517: a startup env var that sets the idle-unload TTL without the settings UI. Unlike the stored setting (gated on auto-switch), the env value is a standalone default that enables idle-unload even with auto-switch off, for headless/container deploys. An explicit UI/API value still overrides it and stays gated. The settings GET reflects the env default when nothing is stored. * Studio: auto-switch fixes from review (paths, embeddings input, env idle reload) - /v1/models advertises a client-facing alias instead of a filesystem path: the ./models and LM Studio scanners report the on-disk path as the model id, so the index now prefers model_id/display_name as the advertised/override id and keeps the concrete path internal as load_path, still resolvable by path. - /v1/embeddings validates input before auto-switch: a request with a model but no input now 400s before the hook (like chat/responses/messages), so an invalid embeddings request cannot unload or swap the resident model. - Standalone UNSLOTH_MODEL_IDLE_TTL reloads the freed model: the hook now runs when auto-switch or idle-unload is active, and with auto-switch off it skips the resolver and only restores the idle-unloaded model, so the first idle timeout no longer leaves later /v1 requests with nothing loaded. - Do not resurrect a stale GGUF over an active Unsloth model: the reload-stash path bails when a non-GGUF backend is loaded, so an unknown /v1 name cannot tear down a live Transformers/Unsloth model. - Defensive HF cache scan: each cache root's resolve/dedup is wrapped so a missing or malformed root skips that root rather than aborting the index. - Single-model retrieve checks the id is a string before lowercasing. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: fix automatic-load asymmetry, audio reload, preview, idle timer The standalone UNSLOTH_MODEL_IDLE_TTL reload is a second automatic-load trigger, but several validate-before-switch guards and reload hooks only checked the auto-switch toggle. Add a shared _automatic_model_load_may_run() (auto-switch on, or idle TTL > 0) and route every guard through it. - /v1/completions validates prompt before any automatic load (it was the one model-bearing route with no pre-check). - /v1/chat/completions and /v1/embeddings pre-checks gate on the shared predicate so a standalone idle TTL cannot reload then reject. - /v1/messages no longer 503s before the reload hook can restore an idle-freed model when auto-switch is off. - Raw completions/embeddings with no model field pass a non-empty sentinel so the idle-stash reload runs, restoring the legacy "omit model, use loaded" path. - /api/inference/audio/generate gains the reload hook (after message validation) so an idle-freed audio GGUF is restored. - Public preview opts out of auto-switch via a request-scope flag, so a caller's model field cannot swap away from the pinned checkpoint; preview chat streams are now matched by _is_inference_path so idle-unload cannot kill them. - Keep-warm no longer stamps activity on request start, and external-provider untracking decrements without restamping, so periodic external traffic can no longer keep the local GGUF warm forever. Merges origin/main (the branch had fallen behind, which also brought in the preview route the review flagged). * Studio: surface model auto-switch in the API tab and demo it in examples The OpenAI auto-switch toggle previously lived only in Settings -> General. Add the same toggle to the API tab's usage-examples panel (it shares the settings cache), and make the examples reflect it: when on, the Python examples append a second call naming a different downloaded GGUF (so the model field visibly selects which model serves), and the curl examples gain a one-line note. Reuses the existing settings API client and i18n keys. * Studio: harden OpenAI auto-switch reload-only path and Anthropic tool validation - Omitted-model raw-body requests pass a reload-only sentinel so the idle-stash reload still restores an idle-freed model, but the resolver never matches a downloaded GGUF literally named "default". - Reject malformed Anthropic client tools before _maybe_auto_switch_model so an invalid request can no longer evict the loaded model. * Studio: extend auto-switch reload-only and tool validation to schema endpoints - Schema-backed endpoints (chat completions, responses, count_tokens, messages, audio) defaulted an omitted model to "default" and passed it to the switch hook, so a downloaded GGUF named "default" could be swapped to. Route the hook through a helper that switches only on an explicitly set model, else reload-only. - Propagate the explicit-set status when building the chat request from a Responses request, so the non-streaming chat re-check stays reload-only too. - Validate Responses function tools before the switch hook so a malformed tool returns 400 without evicting the loaded model. * Studio: serialize auto-switch swaps across event loops with a process-wide gate The auto-switch lock is a per-event-loop asyncio.Lock, so two /v1 swaps on different loops in one process could both pass it and race the single model slot (the backend and _load_model_impl are process-wide). Add a process-wide threading gate around the swap, acquired off the loop so a cross-loop wait never blocks it, layered with the existing per-loop lock. Add a cross-loop test that fails without the gate (two slow loads overlap) and passes with it. * Studio: make the auto-switch swap gate wait cancellation-safe _acquire_swap_gate awaited asyncio.to_thread(lock.acquire) when another loop held the process-wide gate. to_thread cancellation doesn't stop the worker thread, so a /v1 request cancelled mid-wait (client disconnect during a cross-loop swap) would have its thread acquire the gate after the fact, while the finally that releases it never runs -- permanently deadlocking later auto-switch swaps. Poll a non-blocking acquire off a short asyncio.sleep instead: it still keeps the wait off the loop and serializes across loops, but a cancel now lands during the sleep, when the gate is not held, so nothing leaks. Add a test that deadlocks the to_thread variant (it times out) and passes with the poll. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: validate modality and tool-confirmation before auto-switch Two more request shapes could load a named GGUF and only then 400, evicting the resident model: - An image request naming a different text-only GGUF. The switch hook now takes require_vision and rejects a swap to a non-vision target before loading it; a GGUF's vision capability is its companion mmproj, knowable without a load, and matches the post-load guard. Only the resolver branch is checked, never the reload-stash restore. - confirm_tool_calls=true with stream=false and local tools. /v1/chat/completions now rejects that shape before the hook, mirroring the local tool path's bypass_permissions exemption and intent signal. The vision probe threads the ambient HF token to keep the capability-probe invariant. Reload-only and idle-reload paths are unaffected. * Studio: extend validate-before-switch and make the lifecycle gate process-wide - /v1/messages/count_tokens now rejects malformed client tools before the switch hook, like /messages (shared _validate_anthropic_client_tools helper), so a count request can't evict the loaded model. - /v1/chat/completions rejects a malformed tool_choice forcing object (a {"type":"function","function":{}} with no name) before the switch hook. - The inference lifecycle gate that blocks new inference during a swap is now process-wide (a poll-acquired threading lock, cancellation-safe), not a per-loop asyncio lock, so a request on another event loop can't start inference while a swap tears the single backend down. - Usage examples no longer hard-code a switch-demo repo most users lack; the model is an explicit placeholder the user replaces. * Studio: extend the auto-switch modality guard to /v1/responses and /v1/messages The pre-load vision check that guards /v1/chat/completions now also runs on /v1/responses and /v1/messages, so an image request naming a text-only GGUF is rejected before the swap and never evicts the resident vision model. Run the vision capability probe off the event loop. Make the /v1/models retrieve loaded fast-path case-insensitive, and never advertise a host path from the resolver. Remove the dead list_switch_eligible_ids helper, superseded by the /v1/models catalog. * Studio: filter /v1/models to GGUF, per-loop catalog lock, reject system-only Responses Address review findings on the auto-switch path: - /v1/models advertises only GGUF models the API can actually switch to; a safetensors/LoRA entry would be selectable but never loadable via llama.cpp. - The /v1/models catalog cache uses a per-loop lock (like the auto-switch path) so a second event loop awaiting it can't hang in a multi-loop process. - /v1/responses rejects system/developer-only input before the switch, mirroring chat, so an invalid request can't evict the resident model. - _build_index guards each scan source on its own so one bad root drops only that source; the vision probe logs a real detection failure instead of swallowing it. * Studio: list cached GGUFs in /v1/models by inspecting files, not model_format The HF-cache scanner leaves model_format unset for GGUF snapshots, so the previous model_format == "gguf" filter dropped every downloaded HF-cache GGUF from /v1/models and the retrieve fallback. Decide GGUF-ness from the on-disk files via the resolver (info_has_local_gguf) instead, run off the event loop, so the catalog advertises exactly what /v1 can serve. * Studio: fix /v1/messages/count_tokens route binding plus auto-switch review fixes The @router.post decorator for /messages/count_tokens had been separated from anthropic_count_tokens by the _validate_anthropic_client_tools helper, so the route bound to the validator and dropped its auth dependency. Move the decorator back onto the handler. Add route-binding tests asserting each /v1 endpoint maps to its handler with the auth dependency, so a decorator/handler split is caught at the route level (the direct-call tests missed it). Also from review: - update_openai_auto_switch writes both settings keys in one transaction so a PUT can't leave one updated and the other stale (drop the now-unused single setters). - max_seq_length override rejects 0 at the boundary (ge=1) instead of accepting then silently dropping it. - Document that embeddings auto-switch is best-effort: GGUF pooling has no cheap pre-load probe like vision's mmproj, so a guard would false-reject GGUF embedders. - Add a positive idle-unload test (loop frees the model and stashes it for reload). * Studio: validate Responses tool_choice + Anthropic mixed tools before switch, filter Ollama from catalog More auto-switch review findings: - /v1/responses rejects a forcing-function tool_choice with no name before the switch, mirroring chat, so a malformed request can't evict the resident model. - /v1/messages rejects mixing Anthropic server tools with custom client tools before the switch (the check depends only on the payload, so it moves up cleanly). - /v1/models no longer advertises Ollama-link models: info_has_local_gguf excludes .studio_links / ollama_links entries, which the resolver skips and can't switch to, so an advertised id never silently falls through. * Studio: guard chat audio input before switch; surface env-backed idle unload in settings UI A chat request carrying audio_base64 rides the same companion mmproj projector as a vision request, so a text-only target cannot serve it either. Flag require_vision for audio input as well so the multimodal probe runs before the switch and a rejected request never evicts the working model. Generalize the reject message to cover image and audio. The settings response now reports idle_unload_active (effective TTL > 0) so the UI can distinguish idle-unload that is active via the UNSLOTH_MODEL_IDLE_TTL env var from the case where it needs the toggle enabled. * Studio: harden auto-switch eviction guards (count_tokens vision, TTS reload-only, mmproj/stash) Four eviction/correctness fixes on the opt-in /v1 auto-switch path: - /v1/messages/count_tokens now carries the same require_vision guard as /messages, so an image count naming a text-only GGUF can't evict a loaded vision model for a swap that can't serve the request. - /audio/generate is now reload-only. A local GGUF's audio-input capability is not a cheap pre-load probe (the companion mmproj signal can't tell an audio projector from a vision one, and codec TTS ships no projector), so resolving the client model could load a text/vision-only target and evict the working audio model before the audio check fails. Only the idle-stash restore runs here; switching TTS models is an explicit /load. - The resolver no longer treats a standalone mmproj .gguf as a servable model. _scan_models_dir's standalone-file pass does not filter mmproj the way its directory scan does, so /v1/models could advertise a projector and a switch could load it over the real weights. - A non-GGUF (Transformers/Unsloth) load and a deliberate /unload now clear the idle reload stash, so a manual load/unload is never superseded by a stale idle-freed GGUF that the next /v1 request resurrects. * Studio: report advertised repo id consistently after an auto-switch Two model-id reporting fixes so an auto-switched cached HF GGUF is named by its repo id everywhere, not its snapshot path: - Streamed /v1/responses envelopes now derive the model id from _llama_public_model_id (which prefers _openai_advertised_id) instead of the raw model_identifier. After an auto-switch the identifier is the snapshot path while the repo id lives in _openai_advertised_id, so the stream used to report a snapshot basename while /v1/models, chat completions, and non-streaming Responses all reported the repo id. - When an advertised alias already resolves to the loaded model (a model loaded by local path, requested by its repo or LM Studio id), the already-serving early return now records the alias as the advertised id, so /v1/models and responses report the alias and mark it loaded instead of the path-derived basename. Resolver branch only; safe lock-free because an in-flight request blocks any concurrent swap via the single-slot busy guard. * Studio: validate request shapes before auto-switch (prompt/input/audio/mcp confirm) Four more validate-before-switch guards so a deterministic client error never evicts the resident model on the opt-in /v1 auto-switch path: - /v1/completions rejects an object/number prompt (only a string or array is valid) before the switch, instead of loading the named GGUF and letting llama-server reject the shape afterward. - /v1/embeddings rejects an object/number input the same way. - Chat rejects an oversized audio_base64 upload (413) before the switch. The size cap is a cheap, target-independent length check; the decode itself stays post-switch to avoid decoding a valid upload twice. - The chat confirm-without-stream pre-switch guard now mirrors the tool loop's actual enablement: _effective_enable_tools (honoring a CLI --enable-tools policy) and mcp_enabled (which opens the tool loop on its own but defers to a CLI --disable-tools policy). Previously a confirm+no-stream request with only mcp_enabled slipped past and 400'd after the swap. * Studio: fix model-id retrieval, streaming n>1, resolver cache TTL, keep-warm auth Four fixes from review: - GET /v1/models/{id} legacy raw-path fallback now maps the raw identifier to the same public id its /v1/models entry uses. After an auto-switch load the identifier is the snapshot path while the entry is keyed by the advertised repo id, so a client that cached the old absolute path no longer 404s on a model that is in fact loaded. - stream=true with n>1 is now rejected before the switch. Only the non-streaming GGUF path returns multiple choices, so streaming n>1 is invalid on every local serving path; both fields are known pre-switch, so it must not load model B only to 400 and evict model A. Non-streaming n>1 stays post-switch where the serving path decides. - The resolver index cache is stamped after _build_index, not with the pre-scan timestamp. On installs with enough local models for the multi-root scan to exceed the 5s TTL, the cache was stored already expired and every request rebuilt it. - The keep-warm middleware no longer stamps model activity for 401/403 responses. It runs before FastAPI auth, so unauthenticated probes used to refresh the idle timer without touching llama.cpp; they now decrement the in-flight count without keeping the model warm. --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: oobabooga <112222186+oobabooga@users.noreply.github.com>	2026-07-01 06:42:23 -07:00
Daniel Han	e8945cab46	Whole-document context for RAG chat attachments (#6693) ... * Add whole-document context mode to RAG chat attachments Thread-attached files are injected in full when they fit a token budget, instead of only top-K retrieved chunks, so the model reads the entire file for summarize/reason-over-document requests. Oversized files fall back to top-K retrieval so the context window is never blown. KB and project corpora are unchanged (still retrieval). - core/rag/store.py: all_chunks_for_scope returns every completed-document chunk for a scope, ordered document-then-index, joined with filename. - core/rag/tool.py: whole_document_context renders the chunks as the same <chunk> blocks + citation source-map retrieval produces, returns None when empty or over budget. - core/inference/tools.py: build_rag_autoinject tries whole-document first for thread scopes, falls through to search_for_autoinject otherwise. - core/rag/config.py: THREAD_WHOLE_DOC + WHOLE_DOC_MAX_TOKENS (env-tunable). - tests/test_rag_whole_document.py: store ordering, whole-doc render + budget cutoff, auto-inject whole-doc vs top-K fallback, KB never whole-doc. * Add scanned-PDF OCR fallback to RAG ingestion A PDF page with no extractable text layer (a scanned or image-only page) previously ingested as empty, so image PDFs were invisible to retrieval and whole-document context. Such pages are now rendered and transcribed by the loaded vision model during ingestion, so they become searchable and readable like any other page. This restores OCR for the RAG document flow without a separate extraction pipeline. - core/rag/parsers.py: render_pdf_pages renders whole pages (1-based) to PNG. - core/rag/captioner.py: factor the shared vision call into _vision_complete; add _ocr_one + ocr_pages (transcribe rendered pages, OCR_MAX_PAGES bound). - core/rag/ingestion.py: _ocr_scanned_pages runs right after parse, replacing text on near-empty PDF pages. No-op when OCR is off, no page is scanned, or no vision model is loaded (degrades like figure captioning). - core/rag/config.py: OCR_SCANNED, OCR_MIN_CHARS, OCR_MAX_PAGES, OCR_DPI, OCR_TIMEOUT_S, OCR_MAX_TOKENS (env-tunable). - tests/test_rag_ocr_fallback.py: page render, ocr_pages gating + cap, scanned PDF end-to-end OCR into chunks + whole-doc, born-digital skips OCR, disabled leaves the page empty. * Broaden OCR prompt to figures/tables and guard against repetition runaway The OCR prompt now asks the vision model to also transcribe text inside figures, diagrams, charts and tables, so labels and table cells on scanned pages are indexed rather than skipped. Verified on real documents that this does not regress plain-text transcription. Some vision models loop on sparse images (e.g. a title-only cover) and emit the same line hundreds of times. _collapse_runaway caps any run of identical consecutive lines so a pathological page cannot flood the index; legitimate short repeats (a label appearing a few times) survive. Applied in ocr_pages. * Restrict whole-document injection to thread attachments only whole_document_context resolved the combined project+thread scope, so a project chat (the frontend sends both thread_id and project_id) injected the entire project corpus in full, contradicting the design that project and KB corpora stay retrieval-only. A large project corpus could also push the total over budget and drop a small thread attachment back to top-K. Resolve the thread scope alone in whole_document_context, and in build_rag_autoinject only enter whole-doc mode when a thread attachment is present and no KB is selected (a KB pick is exclusive: search that corpus). Project sources and KBs keep top-K retrieval. Adds regression tests for the mixed project+thread payload, the budget isolation, and KB precedence. * Address review: keep project retrieval, harden budget + OCR guards Follow-up to the 8-reviewer pass on the whole-document + OCR work. - Preserve project grounding in project chats. The thread-scope-only fix made whole-doc exclusive of retrieval, so a thread attachment silently dropped the project corpus for that turn. build_rag_autoinject now whole-docs the thread attachment AND retrieves the project sources top-K, merged under one citation numbering via tool.render_sources. KB selection stays exclusive. - Budget: a NULL/zero token_count no longer bypasses the cap (length-based fallback in _row_token_count), so a malformed huge doc can't inject in full. - OCR runaway guard: _collapse_runaway now also caps each distinct line at a generous total across the page (not just consecutive), bounding the interleaved/alternating loops weak models emit; blank-line floods collapse too. - OCR: warn when a scanned PDF exceeds OCR_MAX_PAGES (pages past the cap stay untranscribed) instead of silently dropping them. - Document the known limits: OCR'd pages have no PDF highlight regions; vision models need a micro-batch >= image tokens (Gemma-family) or the server aborts. - Tests for project-retrieval composition, NULL-token budget, and interleaved runaway; drop the now-superseded exclude-project test. * Add OCR toggle to RAG retrieval settings Make scanned-PDF OCR user-controllable per upload instead of only via the RAG_OCR_SCANNED config default. The retrieval settings panel gains an OCR scanned pages switch (persisted in localStorage, on by default); the chosen value is read fresh at upload time and sent with each document upload. Backend: the three upload routes accept an optional ocr form field and pass it through start_ingestion to _ocr_scanned_pages, which now treats None as use the config default and an explicit bool as an override. The on/off policy lives only in _ocr_scanned_pages now, so ocr_pages no longer re-checks the config (that double gate would have blocked a per-upload ocr=True while the default was off). Tests cover both override directions (force on while config off, force off while config on). * Add "Describe figures & charts" toggle with chart-aware captions Surface RAG figure captioning as a user control and make it actually useful for graphs and plots. The figure detection already clustered vector drawings and raster images into regions and rendered them, but captioning was off by default, had no UI, and used a thin generic prompt. Accuracy: the caption prompt now asks for chart type, axis titles and units, legend or series, salient trends and readable values, and table columns, while forbidding invented numbers. The token budget is configurable (CAPTION_MAX_TOKENS) and captions pass through the same runaway guard as OCR so a looping vision model cannot flood the index. Control: a per-upload caption override threads from the three upload routes through start_ingestion and _run, with the on/off policy single-sourced in _run (caption self-gating removed from caption_images, mirroring the OCR change) so a force-on override works when the config default is off. The frontend adds a "Describe figures & charts" switch in the retrieval settings, persisted in localStorage and sent with each upload. Default on; it is a no-op without a vision model and bounded to CAPTION_MAX_IMAGES figures per document. Tests cover the new caption_images contract, the runaway guard on captions, the chart-aware prompt and token budget (and that OCR keeps its own prompt and budget), and both override directions end to end through ingestion. * Generalize figure understanding: transcribe-first prompt + high-DPI tiling Make figure/chart description work across any visual and any model strength, not just a strong VLM on simple figures. Two changes, validated by a recall benchmark on authoritative documents (ResNet/Attention papers, USDA, UN UDHR). 1. Transcribe-first caption prompt. The caption now asks the model to transcribe every visible label verbatim (titles, axis labels and units, legends, every box/node/arrow label, table cells, equations) and then add a one-line summary, instead of only describing the figure. Transcription is the most model-robust visual task, so weak models that cannot reason about a chart still recover its labels. 2. High-DPI tiling of figure pages. Figure-bearing pages are rendered as an overlapping grid of high-DPI tiles (plus a full-page pass for context); each tile is transcribed, then merged and de-duplicated. This keeps small diagram labels legible and covers every sub-figure without relying on exact region detection, which previously missed sub-figures and small labels. Supporting changes: figure render DPI 130 -> 200 with a clip margin so edge labels are not lost; vision calls are deterministic (temperature 0) so transcription does not randomly drop labels; the repetition guard now applies to captions too. New config knobs: FIGURE_DPI, FIGURE_MARGIN_FRAC, FIGURE_TILE_ROWS/COLS, FIGURE_TILE_ OVERLAP, FIGURE_FULLPAGE, CAPTION_MAX_PAGES, larger CAPTION_MAX_TOKENS, and CAPTION_MAX_IMAGES as a per-document tile budget. Measured figure context recall (per-label, dense academic figures): Qwen2.5-VL: 0.50 -> 0.83 (overall 0.81 -> 0.94) Gemma-4-E2B (weak): ~0 with loops -> 0.83 (overall 0.91) Born-digital text and scanned-page recall are unchanged (no regression). parsers gains _figure_boxes (shared detection), pages_with_figures, and render_pdf_figure_tiles; captioner gains merge_page_captions and a temperature parameter; ingestion routes figure captioning through the tiled path. * Fix RAG review issues: whole-doc budget pre-check, figure gating, empty re-ingest, vision auth Whole-document context now runs a cheap token-sum pre-check (store.scope_token_estimate) before hydrating every chunk's text, so an attachment that cannot fit the budget is rejected without loading the whole corpus into memory. The estimate mirrors all_chunks_for_scope's filter and the per-row token-count fallback exactly. Ingestion skips all figure work (PDF rasterization and detection, not just the caption call) unless a vision model is loaded, so a text-only deployment pays nothing. When OCR is enabled, scanned/image-only pages are excluded from figure tiling since OCR already transcribes them whole, avoiding double vision work and overlapping index entries; a scanned figure page is still tiled when OCR is off. start_ingestion no longer dedupes forever to a prior ingest that produced zero chunks (e.g. a scanned PDF uploaded before a vision model was loaded): the empty record is dropped and the content is re-ingested. Vision OCR and caption requests now send the backend Authorization header, so they match the chat endpoint and do not 401 under direct-stream (--api-key) mode. Adds tests for the budget estimate, scanned-page exclusion, the vision-model gate, the empty re-ingest path, and the auth-header passthrough. * Trim RAG vision-ingestion comments and docstrings Tighten the verbose multi-line docstrings and comments added across the RAG vision ingestion work (captioner, config, parsers, ingestion, store, tool, build_rag_autoinject, the RAG tests, and the chat-store/upload-hook frontend toggles) to one or two lines while keeping their intent. No code changed: verified comment/docstring-only against the prior commit, and the RAG test suite still passes. * Fix figure-tiling exclusion and client dedupe for re-ingestable docs Figure tiling now excludes only the pages OCR actually transcribed, not every text-less page. _ocr_scanned_pages returns the set of pages it OCR'd, and _run passes that to pages_with_figures as exclude_pages (replacing the ocr_on-keyed min_text_chars heuristic). A scanned page that OCR skipped (past OCR_MAX_PAGES, or whose OCR returned empty) is no longer dropped from captioning, so a chart on such a page still gets a caption. The document panel's upload dedupe no longer skips re-selecting a file whose only matching doc completed with zero chunks. Such a doc is re-ingestable (e.g. a scan attached before a vision model loaded), and the backend re-ingests on the same content hash, so the client must let it reach the backend; healthy or still-indexing docs are still skipped. The SSE complete frame's chunk count is recorded on the doc so the check is exact. Adds a regression test for the un-OCR'd scanned figure page and updates the pages_with_figures test to the exclude_pages interface. * Address review findings: whole-doc budget guard, job numChunks, dead code, upload cap whole_document_context now treats a non-positive max_tokens as "never inject" instead of injecting the whole corpus unbounded, so RAG_WHOLE_DOC_MAX_TOKENS=0 tightens rather than disables the budget (the real off switch stays RAG_THREAD_WHOLE_DOC=0). The job-status endpoint and get_job_status now expose num_chunks (joined from the document), and the upload hook threads it through the SSE-fallback completion paths (reconcile + poll). Previously a document that finished via the connection-cap fallback had no chunk count client-side, so the re-ingest dedupe wrongly treated it as empty and re-uploaded it. IndexJob/JobEvent gain the field and the untyped cast is dropped. Removes the dead render_pdf_figures function (superseded by the tiling path), its test, and the unused FIGURE_MARGIN_FRAC config knob. Adds an upload size cap (RAG_MAX_UPLOAD_BYTES, default 200 MB; 413 on exceed with the partial file cleaned up) so a pathological file can't drive unbounded parse + vision work. render_pdf_figure_tiles clamps rows/cols to >= 1 (no ZeroDivisionError on a misconfigured grid). Captioning progress is reported after OCR so the bar is monotonic. sqlite connections set busy_timeout=5000 so a long figure/scan ingest holding its connection doesn't make a concurrent ingest/read fail with "database is locked". Adds tests for the non-positive budget, the zero-grid clamp, job-status num_chunks, and the oversize-upload rejection. * Extract PDF text as layout-aware Markdown via pymupdf4llm parsers._pdf now extracts each PDF page as Markdown with pymupdf4llm.to_markdown (page_chunks=True) instead of flat page.get_text("text"), so tables, headings and lists keep their structure in the indexed chunks and retrieve far better (a table's cells stay associated with their row instead of flattening into a token stream). Gated by RAG_PDF_MARKDOWN (default on); falls back to plain PyMuPDF text when the toggle is off, pymupdf4llm is missing, extraction fails, or a page yields no Markdown. The scanned-page OCR and figure-tiling passes operate on rendered pixels and are unaffected; docx/html/txt keep their existing extractors. The preview-highlight locator already strips Markdown punctuation when building anchors; it now also splits anchor tokens on pipes so a Markdown table row still anchors to the raw PDF word stream. Declares pymupdf4llm as a studio/RAG dependency (was only transitively present via the data-designer plugin). Adds parser tests (Markdown table reaches the page text, the plain-text fallback, the missing-lib fallback) and a locator test for table-pipe anchoring. * Pin pymupdf4llm to 0.3.4 so the package scan does not pull onnxruntime The lockstep pymupdf4llm 1.27.x line makes pymupdf-layout a hard dependency, which in turn pulls onnxruntime (plus numpy/networkx/protobuf). The security-audit pip scan-packages job resolves requirements --with-deps, so adding pymupdf4llm to no-torch-runtime.txt and studio.txt surfaced onnxruntime's un-baselined CRITICAL finding and flipped the hf-stack shard from pass to fail. pymupdf4llm 0.3.x keeps pymupdf-layout behind an optional [layout] extra, so a plain install resolves to pymupdf + tabulate only and never touches onnxruntime. 0.3.4 requires pymupdf>=1.27.1, satisfied by our pinned pymupdf==1.27.2.3, and to_markdown (page_chunks=True) produces equivalent layout-aware Markdown on real PDFs (verified on the Attention, ResNet and USDA documents). Production already installs these files --no-deps, so onnxruntime was never shipped at runtime; this only fixes the scanner. The parser test now asserts Markdown markup (heading or table pipes) rather than table pipes specifically, since 0.3.4 emits a heading but not a pipe table on the tiny borderless synthetic fixture; both markers are absent from the plain-text fallback. * Fix RAG whole-doc review findings * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Address RAG whole-doc review follow-ups * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Address RAG review follow-up edge cases * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Reserve image budget for whole-document RAG * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: danielhanchen <michaelhan2050@gmail.com> Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com> Co-authored-by: wasimysaid <wasimysdev@gmail.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-06-30 15:55:23 +02:00
Michael Han	11469a60fe	(feat) Add project names to studio training runs (#6512) ... * (feat) Add project names to studio training runs to avoid models being overwritten when doing similar training runs * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update studio/frontend/src/features/export/export-page.tsx Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * Update studio/frontend/src/features/export/export-page.tsx Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * Update studio/frontend/src/features/export/export-page.tsx Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * better project name sanitization, removed duplicated project name normalization * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * implement checkpoint scanning utilities and tests for base model inference * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Guard project_name against null and use leading important modifiers * Fix/adjust training project names for PR #6512 * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix/adjust training project names for PR #6512 * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Address project-name review feedback * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Show project names in training recents * Keep GGUF export directories source-specific --------- Co-authored-by: NZ-Linix <nz-linix@outlook.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: NZ-Linix <linus.ordowski@outlook.com> Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: wasimysaid <wasimysdev@gmail.com>	2026-06-29 16:06:36 +02:00
Daniel Han	2ef394137a	Studio: harden background consumer loops and streaming paths against silent UI freezes (#6653) ... * Studio: harden the data-recipe and inference consumer loops against pump death Follow-up to #6643. The same single-unsupervised-consumer pattern the training pump had lives in two sibling loops, with the same failure mode: one bad event kills the only thread that updates the in-memory state every UI surface reads, while the worker subprocess keeps running. - data_recipe JobManager._pump_loop: a malformed worker log line that makes parse_log_message raise no longer kills the pump. Guard _handle_event, the queue read, and the worker-exit finalize, and broaden _drain_queue so a drain error still finalizes the job instead of leaving it wedged "active" (which also leaked the workflow-scoped API key until its 24h expiry). - inference InferenceOrchestrator._dispatcher_loop: guard the routing body so a malformed response or a mailbox put error can't kill the dispatcher and hang every in-flight generation (callers key liveness on the subprocess, not on this thread). Adds regression tests for both. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: extend consumer-loop hardening to RAG, hub, auth, and stream-reader paths Continuation of the data-recipe and inference pump hardening: the same "background producer updates in-memory state that a single unsupervised consumer surfaces to the UI" pattern shows up in several more Studio paths, each able to silently freeze a UI surface while the worker keeps running. RAG ingestion SSE (core/rag/ingestion.py): - job_events polled the queue with a blocking get and never noticed client disconnect or a dead worker, so a closed tab or a producer that died without emitting a terminal event left the stream hanging. It now polls with a timeout, emits heartbeats, ends on terminal job status, caps idle time, and always pops the job registry in finally. - Added _reap_finished_jobs() and call it from start_ingestion so finished job state does not accumulate. Startup reconcile (storage/rag_db.py, main.py): - reconcile_orphaned_ingestion_jobs() marks ingestion jobs (and their documents) that were left non-terminal by a previous crash as failed, so the UI does not show jobs stuck "running" forever after a restart. Wired in at startup next to cleanup_orphaned_runs(). Hub download watcher (hub/services/download_lifecycle.py): - _watch() could leave a job pinned "running" if finalize raised. Body is now guarded: on failure it logs and sets the job to error, and always invalidates the hf cache scan in finally. External provider stream (core/inference/external_provider.py): - read timeout was None (no stall ceiling); set to 300s so a wedged upstream surfaces as an error instead of an indefinitely hung stream. Auth store (auth/storage.py): - Enable WAL + busy_timeout on the auth DB so token validation (read on every request) and login writes stop serialising on the rollback journal. Matches studio_db / rag_db / providers_db. Login rate limiter (routes/auth.py): - _LOGIN_IP_BUCKETS could grow unbounded under spoofed-IP traffic; cap it and prune stale buckets, mirroring the per-account bucket handling. Training progress SSE (routes/training.py): - Break promptly on client disconnect instead of waiting for the next yield to fail on a closed socket, matching the export / data-recipe SSE routes. llama-server stdout drain (core/inference/llama_cpp.py): - Broaden the drain guard so an unexpected decode/read error logs at debug and stops the drainer cleanly instead of escaping the thread. Frontend stream readers (chat-api.ts, rag-api.ts): - Wrap the SSE read loops in try/finally + reader.cancel() so early return ([DONE]), thrown errors, and consumer aborts release the reader lock instead of holding it until GC. Tests: - test_training_progress_stream_nan: fake request now implements the async is_disconnected() the route polls, matching the other SSE route fakes. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: address Codex review feedback on the consumer-loop hardening Four follow-ups from the automated review, all on code this PR introduced: - Data-recipe pump (manager.py): a queue read that keeps raising an error outside the read's narrow catch set (e.g. a broken queue pipe after the child died) hit the `continue` guard and skipped the dead-worker finalize below, spinning forever and leaving the job wedged "active" with its workflow key unretired. On a read failure, fall through to finalize when the worker is no longer alive. Added a regression test. - RAG ingestion SSE (ingestion.py): the 5-minute idle cap could end the stream while the job was still pending/running (a large document spends minutes in embedding/storing with no per-batch progress event). The route then sends [DONE], and the client treats a no-terminal-frame end as completion, marking the document indexed mid-ingestion. Drop the idle cap: while the worker is alive and non-terminal we keep heartbeating; the stream ends only on terminal DB status, the None sentinel, or client disconnect. - Login rate limiter (auth.py): the per-IP path pruned but then added the new IP unconditionally, so a spoofed-source-IP spray kept _LOGIN_IP_BUCKETS unbounded and made every new IP pay a full-dict prune scan. Gate the add on the cap, mirroring the account path. - Hub download watcher (download_lifecycle.py): if finalize raised before it reaped (proc.wait) and dropped the worker (e.g. an I/O error draining stderr), the crash path published a terminal state while the live Popen stayed registered and kept writing the cache, and the terminal set_job let claim() admit a retry on the same repo. Terminate + drop the worker before setting the terminal state. * Studio: keep login throttling working when the per-IP bucket dict saturates Review follow-up. The previous cap fix skipped creating a bucket for a new IP once _LOGIN_IP_BUCKETS was full, returning ip_fails=0. Under a sustained spray that also fills the account dict, every failure from such an IP then looked first-seen and _login_blocked had no bucket to enforce, so the cap effectively disabled throttling once saturated. Bound the dict with a FIFO eviction instead: if the IP is new and the dict is full, reclaim expired buckets (rate-limited so a burst of distinct IPs can't make each failure an O(n) sweep) and, if still full, evict the oldest-inserted IP. The new IP always gets a real bucket, so a saturating (e.g. spoofed X-Forwarded-For) spray stays throttled while memory stays bounded. Added a regression test that saturates the dict and asserts a later IP is still blocked. * Studio: address Codex review (RAG queue lifecycle, stream error, orphan chunks) Three follow-ups on the Phase 6 changes: - RAG ingestion SSE (ingestion.py): job_events removed the per-job queue in its finally on ANY exit, including an early client disconnect while the worker is still running. That dropped the worker's later events (the queue is the only one _emit writes to) and made a reconnect find no queue and receive only [DONE], which the client treats as completion. Only drop the queue on a terminal exit (None sentinel / terminal DB status); leftover terminal queues are still swept by _reap_finished_jobs. Added queue-lifecycle tests. - External provider stream (routes/inference.py): once the 300s read timeout can fire, the stream's except path failed the monitor but ended without an error frame or [DONE], so the chat client saw a bare EOF and saved the timed-out answer as a successful partial with no error. Emit an SSE error frame (and [DONE]) on stream failure so the client surfaces it. - RAG startup reconcile (storage/rag_db.py): marking a half-ingested document failed left its chunks/fts/vec rows intact, and retrieval filters by scope not status, so a failed document could still be retrieved and cited. Purge the document's chunks when reconciling it to failed (the doc row stays for re-ingest). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: release the remaining SSE stream readers (training, data-recipe, export) reviewer.py follow-up. The chat and RAG SSE readers were wrapped in try/finally + reader.cancel(), but the other three readers built on the same response.body.getReader() pattern were left without it: streamTrainingProgress, streamRecipeJobEvents, and streamExportLogs leak the ReadableStreamDefaultReader lock (held until GC) when the consumer aborts, returns early, or a parse/callback throws. Wrap each in try/finally + reader.cancel() (export already had a try/catch, so it only needed the finally). All five frontend SSE readers now release the reader symmetrically. * Tighten resilience comments and docstrings Condense the verbose explanatory comments and internal-helper docstrings added in this branch to shorter, clearer forms. Comment/whitespace only; verified no code changed via AST diff. No behaviour change. * Studio: keep chunks for completed docs during ingestion reconcile Startup reconciliation flips orphaned (non-terminal) ingestion jobs to failed and purges the document's chunks so a failed source can't be retrieved. But it dropped the chunks unconditionally, so a document the worker had already committed as 'completed' before the crash (only its job row left non-terminal) lost every chunk while still reporting 'completed'. That leaves an empty source that retrieval can't return and dedup (status != 'failed') blocks from re-ingest. Only purge chunks when the document UPDATE actually transitions it to failed; an already-completed document keeps its chunks. Adds reconcile regression tests for both the completed-doc and genuine in-flight-orphan cases. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: drop a finished RAG job's queue when the client disconnects job_events kept the per-job queue until it consumed the None sentinel, so a UI that stops on the terminal event (its reader.cancel aborts the stream before [DONE]) left the queue registered until the next _reap_finished_jobs sweep; a batch of uploads followed by idling retained them all. _run writes the terminal DB status before emitting the terminal event, so on generator exit, drop the queue when the job's DB row is already terminal (worker done, nothing to resume) and keep it only while the worker is still running. Adds a disconnect-after-terminal-event regression test. * Remove stray async task output files committed by mistake * Studio: harden login IP throttle and end progress stream on disconnect Two Codex review items: Login per-IP throttle: when the per-IP bucket dict saturated, FIFO eviction could drop a still-hot (blocked) bucket, so an IP could flood the dict with distinct (or spoofed) source IPs to push out its own bucket and retry as first-seen. Stop evicting hot buckets; a new IP that can't fit now shares a bounded overflow counter that still trips the per-IP threshold, so a saturating spray stays throttled and no live counter is reset. Progress SSE: on client disconnect the polling loop only broke and fell through to the unconditional final 'complete' frame, so a buffered or proxying consumer could read a still-active run as completed. Return from the generator instead. Adds regression tests for both (spray cannot reset a hot bucket; disconnect while active emits no complete frame). * Studio: shard the login overflow counter and stop cancelling chat stream after [DONE] Two Codex review items: Login throttle overflow: the single shared overflow counter meant that once a saturating spray pushed it past the per-IP threshold, _login_blocked returned 429 for every new unbucketed source IP, before credentials were checked -- a global login denial. Shard the overflow into a fixed array of counters keyed by hash(ip), so a hot shard only throttles the IPs that map to it while a single source's repeated failures still concentrate in one shard and stay throttled. Memory stays bounded and no live bucket is evicted. Adds a regression test that a hot overflow shard does not block an unrelated IP. Chat stream: the reader.cancel() in the SSE finally fired even after a natural [DONE]/EOF. The backend finalizes its api-monitor entry right after yielding the sentinel (the local pass-through finishes after the last yield), so a client cancel there can be observed as a disconnect and mark a completed request as cancelled. Track natural completion and only cancel on an early/abnormal exit. (No frontend unit test: the Studio frontend has no test harness.) * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: give prep-timeout test fakes an is_disconnected method The progress stream now ends on client disconnect (await request.is_disconnected() before falling through to the terminal frame). After merging that into the prep-timeout tests added later on main, their _FakeRequest/_ReconnectRequest must provide is_disconnected or the generator raises AttributeError under CI. * Studio: keep the login overflow throttle when bucket capacity frees up _login_blocked only consulted the per-IP overflow shard while the bucket dict was still at capacity. If a slot freed before the 60s window expired (e.g. another IP's successful login calls _clear_login_bucket), a source counted in a hot shard stopped being blocked and its next failure got a fresh per-IP bucket, resetting the throttle the overflow path exists to preserve. Always max in the IP's shard (shards are empty outside saturation, so it is a no-op in the common case). Adds a regression test that a hot source stays throttled after a bucket frees. * Studio: clear a login IP's overflow throttle on successful login _clear_login_bucket reset the per-IP and per-account buckets on a successful login but not the overflow shard, so after the dict saturated and an IP was counted in overflow, a later successful login left those entries behind and the next failed attempt could immediately return 429. Store overflow entries as (timestamp, ip) so a source is throttled by its own count within the shard (also removing cross-IP collateral within a shard), and drop just that IP's entries in _clear_login_bucket. Adds a regression test that a successful login clears the overflow throttle. * Studio: bound the login overflow shard memory under high-cardinality spray The per-IP overflow tracked failures in a time-pruned deque of (timestamp, ip) tuples, so a spoofed-X-Forwarded-For spray of distinct one-off IPs grew memory and the per-check scan with request cardinality for the whole window -- undermining the bucket cap that exists to bound memory. Replace each shard with a fixed- capacity dict (ip -> [count, window_start]): O(1) lookups, and when a shard is full a one-off IP evicts the lowest-count entry (Space-Saving) so memory is hard- bounded while a persistent attacker keeps a high count and is never evicted. Adds a regression test that shards stay within the per-shard cap under a 5000-IP spray. * Studio: purge chunks for already-failed docs during ingestion reconcile The reconcile chunk-purge was gated on the documents UPDATE actually flipping a non-terminal doc to failed. A doc the worker had already marked 'failed' before the crash (job row left non-terminal) was not re-flipped, so its committed chunks were kept and stayed retrievable/citable, since retrieval filters by scope not status. Purge chunks whenever the document is not 'completed' (failed, in-flight, or gone), preserving the completed-doc carve-out. Adds a regression test. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: don't inherit an evicted IP's count onto a new overflow source When a full overflow shard evicted the lowest-count entry, the new source inherited that count (Space-Saving base + 1). If a shard was saturated with hot entries, an unrelated new IP could land at/over the threshold and be 429'd after a single attempt -- cross-IP collateral despite the per-source-isolation intent. New entries now start clean at count 1; the only cost is that a heavy hitter that is the lowest-count entry in a fully saturated shard can briefly reset, which is preferable to blocking a bystander. Adds a regression test. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: carry overflow failures into a new IP bucket on transition _login_blocked took max(per-IP bucket, overflow shard) rather than combining them, so a source could log (threshold-1) failures in overflow during saturation and, once a bucket slot freed, another (threshold-1) in a fresh bucket within the same window -- roughly doubling the per-IP limit. When a saturated-era IP first gets a real bucket, migrate its windowed overflow count into that bucket (and drop the overflow entry) so the combined failures throttle at the intended limit. Adds a regression test. * Studio: reconcile a completed doc's orphaned job to completed, not failed When a crash left an ingestion job non-terminal after its document was already committed as completed, reconcile marked the job failed. After restart the upload UI has no in-memory SSE queue and falls back to getJob(), which treats a failed job as an indexing failure and removes/toasts a document that is actually searchable. Mark the job completed (keeping its chunks) when its document is completed. Extends the completed-doc reconcile test to assert the job status. * Studio: clamp the overflow failure count migrated into a login bucket A saturated source could accrue an unbounded overflow count, then materialize one deque entry per recorded failure when a bucket slot freed, allocating an arbitrarily large deque under the login lock. Only at-or-above the per-IP threshold matters for blocking, so cap the count there at the record and take sites; the migration is now bounded without weakening the limit. * Studio: keep the RAG job stream alive on a transient status read The heartbeat poll read the job row unguarded; a momentarily-locked DB would raise out of job_events, which the SSE route turns into a terminal error frame, and the UI drops a document whose worker is still running. Treat a failed status read as non-terminal: heartbeat and retry, and keep the queue so a reconnect can resume. * Studio: set busy_timeout before journal_mode on the auth DB Switching journal_mode needs a lock, so if a refresh-token write already holds one, journal_mode=WAL raises SQLITE_BUSY and the shared try leaves the connection on SQLite's default zero lock wait. Set busy_timeout first so the switch waits instead of failing. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-06-26 03:31:33 -07:00
Erildo	e54b6babd6	feat: implement thread forking functionality with associated database… (#5810) ... * feat: implement thread forking functionality with associated database updates and UI components * fix(studio/chat): register fork-count listener even when thread unsaved Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Polish thread fork action menu * fix-studio-fork-project-test-order * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: imagineer99 <samleejackson0@gmail.com> Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>	2026-06-15 14:57:39 +01:00
Michael Han	99704ffe47	Studio: project sources backed by RAG (#6205) ... * Studio: make project sources work with RAG and polish project UI Projects had a disabled Sources tab with an Add sources placeholder. This wires it up end to end on top of the RAG engine: - Add a project scope to the RAG store, ingestion and retrieval - New endpoints: POST/GET /api/rag/projects/{id}/documents - search_knowledge_base resolves kb, project and thread scopes; an explicit KB stays exclusive, project and thread scopes combine - Multi-scope search: FTS uses scope IN (...), vec0 KNN runs per scope and merges by cosine score - Lazy ALTER TABLE adds documents.project_id on existing databases - Deleting a project also removes its indexed sources - Sources tab now uploads with progress chips and drag and drop - Chats inside a project auto-enable retrieval over project sources when the project has indexed documents (cached probe, no Docs pill needed); external providers still never receive rag_scope UI polish: - Rounder project cards with folder icon chip and softer shadow - Project header icon in a rounded chip - Chats/Sources pills and Add sources button without borders * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: match Add sources button shadow to the chat composer in light mode * Studio: round project switcher hover pill and pad the folder icon * Studio: remove border from project sources box * Studio: grey hover on project cards and menu, move search into header, widen page spacing * Studio: shorten sources copy, white header pills with composer shadow, fixed-width search, hub-size page headings * Studio: align project landing blocks to the composer width * Studio: restore muted background and flat look on projects header controls * Studio: darker grey hover on project cards in light mode * Studio: soften project card hover grey * Studio: keep project card menu button visible while its menu is open * Studio: drop focus outlines and rings on buttons and clickable icons, keep input focus styles * Studio: address review feedback on project sources - Remove uploaded files from disk when a project is deleted, confined to the uploads root - 404 project uploads when the project does not exist, matching the KB endpoint - Guard lexical search against an empty scope list - Re-invalidate the project sources probe after uploads and removals settle so a chat sent mid-upload cannot cache a stale negative - Keep keyboard focus rings: only mouse focus drops the Tailwind ring, the browser default outline stays removed * Studio: add a green New badge to the project Sources tab * Studio: unify New pills, fully round with soft emerald fill and no border * Studio: a touch more vertical padding on New pills * Fix project RAG source edge cases for PR #6205 * Fix duplicate RAG upload cleanup for PR #6205 * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com> Co-authored-by: wasimysaid <wasimysdev@gmail.com>	2026-06-12 15:42:51 +02:00
ashzak	aefe904d66	feat(studio): implement S3 dataset loading (completes #5951) (#6222) ... * feat(studio): add S3 dataset configuration foundation (#4539) Add foundational types and configuration for S3 bucket dataset loading: - Add S3Config type to frontend training types - Add S3Config Pydantic model to backend training models - Add "s3" as a DatasetSource option - Add s3Config state and setS3Config action to training config store - Add i18n translations for S3 configuration (English and Chinese) This provides the type definitions and UI text for S3 integration. Full implementation requires boto3 dependency and data loading logic. Refs: #4539 * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Wire S3 config into training pipeline and prevent secrets persistence - Pass s3_config from request into training_kwargs so it flows to training subprocess - Add s3Config to NON_PERSISTED_STATE_KEYS to prevent AWS secrets from being saved to localStorage Addresses code review feedback on PR #5951. * Exclude S3 config from database persistence to protect secrets Filter out s3_config (which contains secret_access_key) from the config_json stored in training_runs table, preventing AWS credentials from being persisted to disk. Addresses P1 security feedback on PR #5951. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Re-raise HTTPException in start_training and defer s3 DatasetSource widening for PR #5951 * Redact s3_config from W&B run config and accept camelCase S3 credential aliases for PR #5951 * feat(studio): implement S3 dataset loading end-to-end Builds the actual S3 loader on top of the hardened #5951 foundation, turning the 501-gated scaffold into a working dataset source. Backend: - Add core/training/s3_dataset.py: lists and downloads supported dataset files (parquet/json/jsonl/csv) from an S3 bucket to a temp dir, using IAM-role or access-key credentials. boto3 is imported lazily (optional dep). - Wire s3_config into UnslothTrainer.load_and_format_dataset (downloads then reuses the existing local-file path) and thread it through worker.py. - Replace the 501 "not implemented" gate with a boto3-availability guard so S3 works when boto3 is present and fails clearly when it is not. - Add boto3 to studio.txt requirements. - Add tests/test_s3_dataset.py (8 tests) covering download/filtering, collisions, missing-boto3, and S3Config camelCase/IAM validation. Frontend: - Widen DatasetSource to include "s3"; add s3_config to the training payload type and mapper; add an S3 validation branch and selectS3Source store action. - Add s3-config-form.tsx (bucket/region/prefix/keys/IAM toggle) reusing the existing studio.dataset.s3.* i18n strings. - Add a Hugging Face / Local / Amazon S3 source toggle in dataset-section; the S3 config card replaces the dataset combobox when S3 is selected. - Fix DatasetPreviewDialog to accept the widened DatasetSource type. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix S3 dataset loader for PR #6222 * Fix S3 dataset edge cases for PR #6222 * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix S3 IAM payload handling for PR #6222 * Block multimodal S3 datasets for PR #6222 --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Daniel Han <danielhanchen@gmail.com> Co-authored-by: Ash <ash@MacBook-Pro.local> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: wasimysaid <wasimysdev@gmail.com>	2026-06-12 14:52:04 +02:00
Daniel Han	8848a310df	Studio: clean-room compact RAG (knowledge bases, hybrid search, fast indexing) (#5910) ... Adds a self-contained RAG stack to Studio: knowledge bases with chunked indexing, hybrid (dense + lexical) retrieval, and an automatic first-pass context inject into chat. Embeddings run through a local llama-server GGUF backend (default unsloth/bge-small-en-v1.5-GGUF) with a sentence-transformers fallback. The chat tool loop gains a search_knowledge_base tool, a per-turn re-search cap, and source citation, layered on top of the shared ToolLoopController.	2026-06-09 21:17:04 -07:00
Daniel Han	187144d4e7	Reduce and tighten code comments and docstrings repo-wide (#6095) ... Trim and tighten code comments and docstrings across the repository. Comment-only: every changed file verified code-identical to main via AST/token comparison.	2026-06-08 23:09:51 -07:00
Daniel Han	8292e699e4	Studio: make code comments and docstrings more succinct (#6029) ... Trim and tighten code comments and docstrings across studio/ Python. Comment-only: every changed file verified code-identical to main via AST/token comparison.	2026-06-08 23:07:28 -07:00
Daniel Han	3ce187da02	Formatting: ruff line-length 100, kwarg-spacing passes, drop blank after short local imports (#6079) ... Raise ruff line-length to 100 and extend the local pre-commit format pipeline (def-signature magic-comma normalization, short multi-line assert collapse, kwarg '=' spacing, blank-line-after-short-import removal, adjacent string-literal / f-string+plain merge, redundant-pass pruning). Every transform re-checks the file AST and is dropped if it would differ; the whole-repo reformat is verified AST-identical per file and idempotent.	2026-06-08 04:24:13 -07:00
Wasim Yousef Said	7381958225	Configurable upload Cap studio (for training) (#5808) ... * studio: cap training dataset uploads * studio: clean up failed dataset uploads * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * studio: raise upload limits to 500MB * studio: make upload limit configurable * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * studio: stream upload routes * studio: split recipe upload caps * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * studio: tighten upload limit handling * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * studio: import settings router directly * studio: polish upload cap setting control * studio: cap settings request bodies * studio: stub settings route in desktop auth test --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-06-02 08:52:19 -07:00
Lee Jackson	e0ff6a1404	Studio: manage chat history with projects (#5725) ... * feat: align project sidebar UX with ChatGPT * feat: align project sidebar UX with ChatGPT * feat(chat): load stored project list * feat(chat): add project sidebar workflows * fix: stabilize project page navigation * fix: projects chat loading * fix: show project chat thread * style: sidebar project spacing and hover clipping * style: add expandable project chat history and move-to-project submenu * feat: polish project sidebar * feat: persist project sandbox paths * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix: only create sandbox project workspace dir * feat: add optional project workspace deletion from delete dialog * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix: stabilize chat projects CI failures * fix: polish project chat navigation * Studio: manage chat history with projects Group chats into projects with a dedicated projects page and route. Sidebar shows recents with per-row actions and a vertical more-vertical menu, and the sidebar scrollbar stays hidden so rows never shift on hover. Includes chat settings and composer refinements. * Studio: projects sidebar and breadcrumb polish Sidebar: - Remove the Compare nav item. - Widen the sidebar to match the projects layout. - Replace the scroll-gated bottom fade with a static fade pinned above the profile box, so it no longer attaches to Recents or lags the collapse and expand animation. Topbar breadcrumb (chat-page): - On a project landing show "Projects" linking to the projects list. - Inside a project chat show the project name and chat title, with the project name linking back to that specific project page. - Drop the divider between the model selector and the breadcrumb. * Studio: make project workspace delete test cross-platform test_chat_project_delete_files_removes_workspace rooted the project under pytest tmp_path, which resolves to /private/tmp on macOS. The workspace delete guard refuses paths under the system denylist by design, so the test passed on Linux CI but failed on macOS. Add a workspace_projects_home fixture that keeps tmp_path on Linux and Windows (CI unchanged) and falls back to a home subdir only when the temp root is on the platform denylist. Derive the workspace path from the created project so it tracks the projects home. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: satisfy import-hoist check for new path re-exports documents_root and project_workspaces_root are re-exported from utils.paths but only referenced as __all__ string literals, which the import-hoist safety net does not count as a use. It flagged the two newly added re-exports as unused imports and failed Source lint. Name-load both via a module-level _REEXPORTED tuple so the check sees them used. No behaviour change; consumers still import them from utils.paths. * fix: avoid projects empty-state flash * fix: batch chat search indexing * Studio: polish chat sidebar, run settings, and search - Use the native OS scrollbar for the chat sidebar, Run settings panel, and chat search list instead of a custom scrollbar - Highlight the active run in the sidebar and keep chat search available during training - Stop the training log view from replaying when navigating back to a run - Rename the chat settings panel to Run settings and align its toggle icon and position - Tighten heading and sidebar letter spacing and lighten the Train and Recents labels - Match the search dialog corner style across light and dark and drop the stray border - Make the MCP Servers section header plain text instead of a link - Remove a stray .orig backup file * studio/frontend: restore Compare entry point in the sidebar The chat-projects sidebar redesign dropped the Compare nav item and moved it to thread-sidebar.tsx, which is not imported or rendered anywhere. That left no way for a user to start a new model comparison (enterCompare only fired from the guided tour and the training handoff), and broke the Compare/Recipes/Export UI smoke test that clicks [data-tour="chat-compare"]. Re-add the Compare NavItem to the New Chat / Search group, carrying data-tour="chat-compare" and the same new-comparison navigation as before. * studio/frontend: use Unsloth green for the fallback profile avatar Switch the initials-avatar background from blue to #14b789 so the sidebar and edit-profile avatar match the Unsloth brand colour. * studio/frontend: turn project breadcrumb into a project switcher dropdown * studio/frontend: stop project card kebab clicks from opening the project * studio/frontend: hide project switcher outside projects * studio/frontend: stabilize project switcher loading * style: project switcher alignment --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: shimmyshimmer <107991372+shimmyshimmer@users.noreply.github.com> Co-authored-by: Unsloth <michaelhan@Michaels-MacBook-Pro.local> Co-authored-by: Roland Tannous <rolandtannous@gravityq.ai> Co-authored-by: Roland Tannous <115670425+rolandtannous@users.noreply.github.com>	2026-06-01 22:09:16 +04:00
Nilay	9a907a8acb	Studio: add remote MCP server support (#5750) ... * added remote MCP server support * trim * added tests * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * increased timeout * disabling MCP chat toggle * Fix MCP OpenAI function-name validation + cancel propagation for PR #5750 OpenAI requires function.name to match ^[a-zA-Z0-9_-]{1,64}$ before streaming starts. The existing 64-char length check is necessary but not sufficient: MCP servers can return tool names containing '.', '/', spaces, etc. that would 400 the whole chat request. Validate the composed mcp__<server_id>__<tool> name against the regex, skip + warn on miss, and drop duplicate tool names from the same server (which would also 400 the request as "duplicates"). Also propagate the agentic-loop cancel_event into MCP tool execution so a /cancel POST during a long-running MCP call (e.g. GitHub MCP search across a large repo) actually interrupts the in-flight HTTP call instead of waiting out the 300 s timeout. The watcher polls the threading.Event at 50 ms cadence inside the asyncio loop (matches routes/inference.py's existing cancel-watcher cadence) and races against the call task with asyncio.wait FIRST_COMPLETED. Tests added: - test_mcp_specs_skip_invalid_openai_function_names: drops bad chars - test_mcp_specs_skip_empty_tool_name - test_mcp_specs_drops_duplicate_names - test_call_tool_sync_respects_pre_set_cancel_event Also fix test_desktop_auth.py's router stub that listed every existing router but missed mcp_servers_router, so importing main.py fails after this PR adds it to routes/__init__.py. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * PR #5750 round 2: OAuth cleanup on delete/url-change + mcp_enabled standalone Round 2 of cross-platform validation surfaced two more P1 findings: 1. OAuth tokens never get cleared. fastmcp keys tokens by MCP URL, not by server row, and delete / URL change / use_oauth toggle only updated the SQLite row. Re-registering the same URL would silently reuse the old account's credentials. Adds clear_oauth_tokens_async() in mcp_client.py and calls it from the delete + put route handlers when the row had use_oauth=True and either the URL changes or OAuth is turned off. 2. mcp_enabled=true was ignored unless the caller also sent enable_tools=true. The frontend always sends both together so the UI path was fine, but a direct API caller sending only mcp_enabled would silently get no MCP tools, which contradicts the field's documented "append tools from every enabled MCP server" behavior. Loosens the use_tools gate in both the GGUF and safetensors paths so mcp_enabled opens the tool loop on its own; when the caller did not also opt into built-ins, the built-in list starts empty. Tests added: - test_clear_oauth_tokens_async_no_op_safe - test_delete_server_calls_oauth_cleanup_when_oauth_was_on - test_delete_server_skips_oauth_cleanup_when_oauth_off - test_update_server_clears_oauth_on_url_change - test_update_server_clears_oauth_when_oauth_disabled 26 backend MCP tests pass; full studio/backend suite 1710 passed locally. Cross-platform CI (Linux, macOS, Windows) green on staging fork. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * PR #5750 round 3: reject null bool updates + /test surfaces 400 Round 3 of cross-platform validation: 1. PUT /api/mcp/servers/<id> would 500 with TypeError when the body explicitly set is_enabled or use_oauth to null. Pydantic accepts None for an Optional[bool] and _changes_from_payload then passed None into mcp_servers_db.update_server, which int(None)d. Reject explicit null at the validation layer with 400 instead. 2. POST /api/mcp/servers/test caught HTTPException under "except Exception", so an invalid URL came back as HTTP 200 with {"ok": false, "error": "400: ..."} instead of a real 400. The create + update paths return 400 for the same input. Move validation outside the transport try/except so it surfaces 400. Tests added: - test_changes_from_payload_rejects_null_is_enabled - test_changes_from_payload_rejects_null_use_oauth - test_test_endpoint_surfaces_url_validation_as_400 * PR #5750 round 4: hyphenated MCP tool names + empty-tool-list gate Round 4 surfaces two more interaction bugs between the new MCP path and existing safetensors tool plumbing: 1. OpenAI accepts ^[a-zA-Z0-9_-]{1,64}$ for function.name, and round 1 widened the MCP regex to that set, so MCP tools can now be advertised as `mcp__srv__list-issues`. But the XML tool-call parser in tool_call_parser.py used `\w+` (no hyphen), so the model could call the tool but Studio could not parse the call. Same in routes/inference.py's `_TOOL_XML_RE` stripper, which would leave hyphenated tool-call XML in the visible content. Both regexes now use `[\w-]+`. 2. safetensors_agentic treats `tools=[]` as "allow all" (documented contract, exercised by test_empty_tools_list_does_not_enforce_allowlist). When a caller sends `enable_tools=true` + `enabled_tools=[]` + `mcp_enabled=true` and MCP discovery returns 0, the resolved tool list is genuinely empty and built-in tools (web_search / python / terminal) could execute via the model's emitted call. Fix at the route gate instead of breaking the documented contract: set `use_tools=False` when the resolved list is empty, in both GGUF and safetensors paths. Existing callers who omit `enabled_tools` still get ALL_TOOLS and are unaffected. Tests added (32 total): - test_tool_xml_parser_handles_hyphenated_function_names - test_tool_xml_strip_handles_hyphenated_function_names - test_safetensors_agentic_empty_allowlist_still_means_allow_all (documents the contract round 4 preserved) 1716 passed locally; cross-platform CI on staging fork still green. * PR #5750 round 5: GGUF allow-list + CLI policy + hyphenated params + cancel race Round 5 of parallel-reviewer aggregation surfaced six additional findings; five are real and fixed here: 1. Hyphenated MCP parameter names (`<parameter=issue-number>`) were dropped by the XML parser's `\w+` regex. Extended to `[\w-]+` in both core/inference/tool_call_parser.py and core/tool_healing.py. The latter is GGUF's own copy of the parser/strip patterns and was missed by round 4. 2. core/tool_healing.py's `strip_tool_call_markup` still used `<function=\w+>` so hyphenated MCP tool-call XML leaked into the GGUF visible content even after round 4 fixed the shared parser. 3+4. `mcp_enabled` re-opened the tool loop even when the operator passed `unsloth run --disable-tools` (CLI policy False). Round 2's `(_tools_on or payload.mcp_enabled)` gate ignored the raw process policy. Now reads `state.tool_policy.get_tool_policy()` and gates mcp_enabled on `_cli_policy is not False`. Applied to both GGUF and safetensors paths. 5. GGUF's agentic loop called `execute_tool(tool_name, ...)` without checking the model-emitted name against the per-request tool list, while the safetensors loop already enforces this. Added the same allow-list check so a model that hallucinates a filtered MCP name or a built-in the caller opted out of returns "not enabled" instead of executing. Bonus P2 fixes: - `call_tool_sync` now checks `cancel_event.is_set()` BEFORE creating the call task, so a pre-set cancellation does not open the HTTP transport. - `clear_oauth_tokens_async` moved the OAuth import + construction inside the protected try block; a fastmcp.client.auth load error used to escape and 500 the delete / update route. NOT fixed (verified false or out of scope): - finding #10 "structured_content vs structuredContent": fastmcp's CallToolResult dataclass uses snake_case (verified live against structured-only tool result; fields are `dict_keys(['content', 'structured_content', 'meta', 'data', 'is_error'])`). - finding #11 "asyncio.run from running loop": call_tool_sync is invoked from `asyncio.to_thread` worker threads which have no event loop; asyncio.run() is safe there. Tests added (37 total): hyphenated param names, tool_healing strip, GGUF allow-list gate, cancel pre-set short-circuit, OAuth cleanup constructor-error swallowing. 1721 passed locally, no regressions. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: danielhanchen <danielhanchen@gmail.com>	2026-05-27 07:01:11 -07:00
Lee Jackson	61ed4cac51	Studio: persist chat history in backend storage (#5272) ... * feat: Persist chat history in backend storage * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Address chat tombstone batching review * fix: update desktop auth routes stub * chat db settings storage * chat db settings routes * chat db settings client * chat db settings store * chat db settings wiring * chat db history storage * chat db settings migration * chat db settings fallback * chat db container metadata * chat db legacy migration fixes * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * chat ci auth background reads * chat auth storage fixes * chat migration final fixes * chat export batch message lookup * chat history review fixes * chat prune sync fix * chat settings hydration retry * gate settings persistence * Scope chat-history rows by subject; fix hijack, clear-confirm, hydrate race Backend storage and routes: - chat_threads / chat_messages / chat_settings carry a NOT NULL subject column with composite PRIMARY KEY (id, subject). Two authenticated identities can no longer see or wipe each other's data. - Pre-existing rows on an existing studio.db migrate under sentinel subject __legacy_unscoped__ via rename + rebuild + copy; single-user installs see no behavior change. - ON CONFLICT(id, subject) DO UPDATE ... WHERE chat_messages.thread_id = excluded.thread_id refuses cross-thread re-parenting via upsert. upsert_chat_message + sync_chat_messages now raise ChatMessageThreadMismatch which the routes map to HTTP 409. - replace_thread_messages rejects body messages whose threadId does not match the URL thread (HTTP 400) instead of silently rewriting them. - DELETE /api/chat requires ?confirm=true, returns row count, logs the subject and count. - upsert_chat_settings_merge does read + deep-merge + write inside a single BEGIN IMMEDIATE so concurrent writers no longer drop each other's updates. The route delegates to this helper. - New POST /api/chat/messages:batch returns {thread_id -> messages[]} for many threads in one HTTP call. Subject-scoped. Unknown ids return empty lists instead of 404 so the sidebar/search caller can rebuild atomically. Frontend: - chat-runtime-store: hydrate-failure catch sets settingsHydrated:true so a transient backend blip no longer permanently disables persistence. setParams bumps inferenceParamMutationVersions unconditionally so a slow hydration response cannot clobber a pre-hydrate user edit. saveSettingsPatch replaces the serial chain with a debounced pendingPatch + deep merge; flush on beforeunload. - chat-history-storage: clearStoredChats returns ClearStoredChatsResult distinguishing backend / legacy / both outcomes. listStoredChatThreadsWithMessages uses the batched fetch (one HTTP call) instead of Promise.all per-thread; legacy Dexie fallback only fires when the batch result is empty. - chat-api: batchListChatMessages with graceful 404 / 405 fallback to per-thread listChatMessages for older servers. - chat-thread-tombstones: store {id, deletedAt} tuples with 90-day GC and a 5000-entry cap so localStorage stays bounded. Back-compat reads pre-fix plain strings. Adds removeChatThreadTombstones (rollback) and clearAllChatThreadTombstones (post-legacy-purge clean-up). - use-chat-sidebar-items: deleteChatItem tombstones synchronously BEFORE the backend round-trip and rolls back on failure (restores pre-PR optimistic UX). 300 ms trailing debounce on CHAT_HISTORY_UPDATED_EVENT plus requestSeq guard so stream-time event bursts produce at most one fetch per quiet window. Tests: - studio/backend/tests/pr5272_sim/ adds 64 regression tests covering schema migration from pre-fix shape, subject scoping, cross-thread hijack, bulk-replace mismatch, clear-confirm, concurrent settings, unicode + 2MB content + SQL-injection-safe binding, chunking boundary at 900 and 901 ids, batched endpoint (multi-subject + 1200 ids + per-thread order), and grep contracts for the frontend patches. test_chat_history_storage.py updated to pass subject. Verified locally on Linux + macOS + Windows GitHub Actions runners (staging fork): 64 pass + 2 from the PR's own backend test on all three OSes. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Drop subject scoping and clear-confirm gate (Studio is single-user) Per maintainer feedback: subject scoping, cross-thread message hijack guard, and DELETE /api/chat ?confirm=true gate are unnecessary because Studio is intentionally single-user (the client already shows a confirm dialog before clear-all). This commit reverts those backend changes and keeps only the non-multi-user pieces from the earlier fix commit: - studio_db.py: restored to pre-fix shape; adds upsert_chat_settings_merge which does atomic read + deep-merge + write under BEGIN IMMEDIATE so two concurrent slider drags cannot drop one another's updates. - routes/chat_history.py: restored; put_settings now calls the atomic merge instead of doing the read-merge-write across three separate connections. Adds POST /api/chat/messages:batch to collapse the sidebar/search rebuild from N round-trips to 1. - frontend/api/chat-api.ts: align batchListChatMessages request and response keys with the backend (threadIds / messagesByThreadId). - tests/test_chat_history_storage.py: add atomic-merge concurrency test, deep-merge nested-key test, and 901-id chunking-boundary test. - Drop the pr5272_sim test directory (those tests covered the reverted subject-scoping/hijack/confirm behavior). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix sidebar delete crash, keepalive on settings beforeunload flush, search rebuild race Two correctness bugs and one perf race surfaced by a fresh code review of the prior fix commit: - chat-api.ts: notifyChatHistoryUpdated was declared as a non-exported function, but use-chat-sidebar-items.ts imports it. The import would fail tsc with TS2305 and at runtime the optimistic-delete and delete-failure rollback paths would both throw. - chat-runtime-store.ts + chat-settings-api.ts + chat-settings-storage.ts: the beforeunload settings flush is now actually keepalive. Without it the browser cancels the in-flight PUT on tab close, so the last slider drag is silently dropped (which is exactly the case the debounce+beforeunload combination was meant to protect against). - use-chat-search-index.ts: rebuilds now coalesce with a 300ms trailing debounce and discard out-of-order responses via a requestSeq guard. Matches the sibling pattern in use-chat-sidebar-items.ts so two rapid CHAT_HISTORY_UPDATED_EVENTs (run-start + run-end save during a turn) cannot land with stale data winning. - chat-thread-tombstones.ts: drop dead clearAllChatThreadTombstones with no call sites; Dexie is never wiped so the function has no use. * fix(studio): protect chat persistence writes * fix(studio): align chat history clear semantics * fix(studio): show partial chat clear feedback * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix(studio): preserve chat persistence fallbacks * fix(studio): harden chat thread persistence checks * Preserve chat message timestamps * Gate chat stream on history save * Make chat thread backfill best effort * Avoid chat message 404 probe * Tighten chat legacy fallbacks * chat: server-side ledger so legacy Dexie import is recoverable The boolean localStorage sentinel (unsloth_chat_legacy_imported_to_studio_db) made importLegacyChatsIfNeeded non-recoverable: deleting studio.db while the browser keeps the flag silently hides every legacy Dexie thread from the sidebar (verified by the 3-GPU validation probe; matches the third review comment on PR #5272). Same trap fires for browser-profile sync to a fresh machine and any other path that wipes studio.db while keeping IndexedDB. Source of truth moves into studio.db itself via a new chat_legacy_import_log table keyed by legacy thread id. The ledger disappears together with studio.db, so the next launch re-runs the import from whatever Dexie still holds. localStorage stays as a per-session perf hint only. Performance, all bounded by the three new fast-paths before any backend work: A) localStorage hint says "imported earlier in this session" -- 0 network, ~0 ms. Covers the warm sidebar mount. B) indexedDB.databases() reports no "unsloth-chat" DB -- 0 network, ~1 ms. Covers every new user who never had the old browser-only Studio (the common case after launch). C) db.threads.count() + db.messages.count() are both 0 -- 0 network, ~5 ms. Covers returning users who migrated long ago and Dexie was never repopulated. Only when all three miss does the code talk to the backend (GET /api/chat/import-ledger -> diff vs Dexie -> existing import path -> POST /api/chat/import-ledger to record what was just imported). Per-thread tracking is enough because Dexie is read-only after this PR; a thread's message set does not grow. Backend deployments that predate the import-ledger routes are handled transparently: the client treats 404/405 as an empty ledger and re-runs the (idempotent via UPSERT) import on next launch. Changes: - storage/studio_db.py: new chat_legacy_import_log table (WITHOUT ROWID, PK on legacy_thread_id) + list_chat_legacy_import_log() + record_chat_legacy_import_log() (idempotent batch UPSERT). - routes/chat_history.py: GET + POST /api/chat/import-ledger with the obvious request/response models. - frontend api/chat-api.ts: listChatImportLedger() (returns a Set for O(1) diff) + recordChatImportLedger(), both with 404/405 fallback. - frontend utils/chat-history-storage.ts: importLegacyChatsIfNeeded gains three fast-paths, ledger fetch on the slow path, and writes the ledger after a successful import. The localStorage helper is unchanged on the surface; it just stops being authoritative. - tests: 5 new test_legacy_import_log_* cases (empty default, record + list round-trip, idempotency, input dedup, empty/null ignore). All 9 pre-existing tests still pass. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Make the legacy-import recovery actually recoverable The previous commit added a server-side ledger to make Dexie -> studio.db import recoverable after a studio.db wipe, but the localStorage perf hint still short-circuited the import gate before the ledger was ever consulted. After a wipe, the hint stayed "true" and the bulk re-import never ran -- the ledger sat empty and only the per-thread lazy materialize-on-continue path restored data. Changes: - Remove the localStorage short-circuit from importLegacyChatsIfNeeded so the ledger is checked on every fresh tab. legacyChatImportPromise keeps the per-session cache; the hint now only matters for the listing paths. - Batch the slow path: one db.messages.where().anyOf().toArray() and one batchListChatMessages() instead of 2N round-trips. At 1k threads this drops a multi-second blocking import to a single request pair. - recordChatImportLedger returns {accepted, inserted, supported}. The localStorage hint is only flipped when supported is true, so old backends (404 / 405 / 501) no longer permanently poison recovery. - Ledger backfill: threads already present in chat_threads but missing from the ledger now get added too, so old-FE-then-new-FE deployments don't redo the diff every launch. - Backend response field renamed recorded -> {accepted, inserted}. accepted is the deduped non-empty input count; inserted is the rows actually new (via INSERT ... RETURNING). Bounded by Field(max_length= 10_000) on the request payload. - Storage helpers renamed: chat_legacy_import_log -> chat_legacy_imports, record_* -> upsert_* to match the existing noun/verb conventions. - DEXIE_DB_NAME exported from db.ts; duplicate constant in chat-history-storage.ts removed. - 3 new route-level tests for /api/chat/import-ledger covering the round-trip, the (accepted, inserted) split, and the 10k payload cap. All 18 chat-history tests pass. --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: shine1i <wasimysdev@gmail.com> Co-authored-by: danielhanchen <michaelhan2050@gmail.com> Co-authored-by: Daniel Han <danielhanchen@gmail.com>	2026-05-22 06:18:05 -07:00
Roland Tannous	9a0d6f80cb	studio: API external provider support for chat (OpenAI, Mistral, Gemini, Cohere, Anthropic, OpenRouter, DeepSeek, custom providers) (#4706) ... * studio: add external provider support for chat inference Adds the ability to connect to OpenAI, Mistral, Google, Cohere, Together, Fireworks, and Perplexity from the Studio chat interface. - Provider configs stored in SQLite (no API keys persisted) - RSA-2048 key pair generated at startup for client-side key encryption - httpx proxy client streams SSE responses in OpenAI-compatible format - New /api/providers routes: registry, CRUD, test, models - /v1/chat/completions routes to external provider when provider fields present - Integration test suite covering CRUD, connection, model listing, and inference - Frontend spec doc with full API contract * remove frontend spec doc from branch * fix auth fixture: handle forced password change on fresh install * fix tests: default port 8000, allow 400 for no-model-loaded * fix: update Cohere models to current (command-r retired Sept 2025) * feat: add OpenRouter as 8th provider * feat: add native Anthropic provider with Messages API translation * fix: correct Anthropic base URL and drop top_p (conflicts with temperature) * feat: add DeepSeek provider (deepseek-chat, deepseek-reasoner) * feat: rename google -> gemini, refresh model list to 2.5 series * feat: remove together, fireworks, perplexity providers * feat: multimodal image support for external providers - Add _build_external_messages() that preserves image_url parts for vision-capable providers instead of stripping them - Update _proxy_to_external_provider() to use new helper - Translate image_url content parts to Anthropic native image format in _stream_anthropic() - Add TestVisionInference pytest class (1x1 PNG smoke test) * test: use sloth photo URL for vision test, add Anthropic remote URL support * fix: update Mistral model to mistral-small-2506 * update mistral default model to mistral-large-2512 * fix gemini vision test: download image as base64 data URI instead of remote URL * add gemini-3-flash-preview as default gemini model * fix gemini truncated reply (max_tokens 16->64) and suppress GeneratorExit on client disconnect * increase vision test max_tokens to 215 * fix GeneratorExit: aclose stream generator before closing httpx client * fix httpcore GeneratorExit: explicitly aclose aiter_lines before response closes * fix duplicate [DONE] and suppress httpcore RuntimeError on Python 3.13 asyncgen cleanup * fix: call response.aclose() before lines_gen.aclose() to prevent httpcore RuntimeError on Python 3.13 * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Potential fix for code scanning alert no. 36: Clear-text logging of sensitive information Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * review: add comments for manual iteration rationale, mask password in test print, clarify Anthropic URL/models support * perf: use shared module-level httpx client for connection pooling across requests * studio: add API provider UI and integrate wiring (#4737) * feat: expose external models in selector and chat settings * feat(chat): wire external providers to backend + RSA key flow - Fetch registry/configs; create/update/delete saved providers - Encrypt API keys (Web Crypto RSA-OAEP) for test/models/chat - External model selection + chat payload (provider_id/type, external_model, encrypted key, optional base URL) - Local storage for keys + provider list; small UX/copy and guardrails * add missing providers-api.ts file by Imagineer99 * fix: address PR review comments — system prompt visibility, retry loop, test logging * feat(studio): encrypt external provider API keys at rest in localStorage API keys for external providers (OpenAI, Mistral, etc.) were stored as plaintext in localStorage, vulnerable to browser extensions and XSS. Add password-derived AES-256-GCM encryption: on login the user's password is used via PBKDF2 (100k iterations, SHA-256) to derive an in-memory encryption key. API keys are encrypted before writing to localStorage and decrypted on read. The derived key is never persisted — cleared on logout, re-derived on next login. Legacy plaintext keys are transparently migrated on first access. Password changes re-encrypt all stored keys. No backend changes required — the existing RSA-OAEP transit encryption is unaffected. * fix: cast PBKDF2 salt to BufferSource for strict TypeScript lib types * fix: persist session password in sessionStorage to survive page refreshes * feat(studio): preserve image parts in external provider chat requests toOpenAIMessage() now returns multimodal content arrays (OpenAI vision format) when messages contain images, instead of always flattening to plain text. This enables vision-capable external providers (OpenAI, Gemini, Anthropic, etc.) to receive user images. The backend already handles image_url content parts in _build_external_messages(). * studio: fix external models selectable in chat-only mode (#4779) * fix: external models selectable in chat-only mode * fix: model selector tabs default to active model kind * Studio: API external provider registry + curated catalogs (HF/OpenRouter) and chat UX (#4787) * fix: external models selectable in chat-only mode * fix: model selector tabs default to active model kind * feat(studio): expand provider registry, curated catalogs, and chat UX - Add Hugging Face, Kimi, Qwen; remove Cohere; reorder registry - model_list_mode curated for HF/OpenRouter; lightweight /models check - API returns default models for curated providers; expose model_list_mode - Frontend: provider logos in model picker, providerType on external models - Chat providers dialog: curated vs remote flows, motion polish - Thread: LayoutGroup + composer motion alignment with app easing * fix(studio): disable Anthropic tool-calling flag and preselect curated defaults * feat(studio): add external provider logos and ApiProviderLogo helper * Studio: Polish API Providers dialog (#4899) * fix: lower verbage in API providers page * fix: fix(studio): tune API Providers dialog width with rem-based responsive caps * feat: add custom provider support (#4902) * fix: replace crypto.subtle with node-forge for HTTP compatibility crypto.subtle is only available in secure contexts (HTTPS/localhost), which breaks provider API key encryption when Studio is accessed over plain HTTP on remote GPU VMs. Switch to node-forge for RSA-OAEP and AES-256-GCM operations — same algorithms, works on any origin. * fix: store provider API keys as plaintext in localStorage Drop AES-256-GCM at-rest encryption for provider API keys. The session-password-derived encryption broke on auto-login via refresh token (password never captured), causing keys to silently vanish. API keys are still RSA-encrypted in transit via node-forge. At-rest encryption in localStorage added no real security since the decryption key also had to live client-side. Removes crypto-storage.ts, session password plumbing, and reEncryptAllKeys. * fix: use max_completion_tokens for OpenAI provider Newer OpenAI models (gpt-4o, gpt-5.x) reject the max_tokens param and require max_completion_tokens instead. Other providers still use max_tokens. * fix: skip empty assistant messages in external provider requests Some providers (Mistral) reject assistant messages with empty content. Filter them out when building the message list for external providers. * Update model-selector.tsx * Update model-selector.tsx * Update model-selector.tsx * Update chat-adapter.ts * Update chat-adapter.ts * Update chat-page.tsx * Update chat-settings-sheet.tsx * Update chat-settings-sheet.tsx * Update chat-settings-sheet.tsx * Update chat-providers-dialog.tsx * feat: polish providers settings form UI * style: polish provider row icon sizing and alignment * style: stabilize provider layout * style: add provider API key visibility toggle * fix: add provider render on empty list * studio/frontend: sync package-lock.json with package.json npm ci was failing because node-forge and @types/node-forge were declared in package.json but missing from the lockfile. Ran npm install to regenerate. * studio/backend: fix backend CI failures for providers router - test_desktop_auth: include providers_router in the routes stub so studio.backend.main imports cleanly under the monkeypatched module - test_providers_api: skip the whole module when STUDIO_TEST_PASSWORD is unset (it is an integration test against a live Studio server, same shape as the already-ignored test_studio_api.py) * studio/chat: drive ChatSettingsPanel from a per-provider capability map Replace the binary isExternalModel toggle in the sampling section with a provider-aware capability map. Each external provider type advertises which of top_k / min_p / repetition_penalty / presence_penalty its chat-completions API actually accepts, so the panel only renders the knobs that map onto the active provider's request body. Anthropic now exposes top_k; DeepSeek hides presence_penalty (deprecated in their docs); OpenRouter and custom providers continue to show every knob (OpenRouter drops unsupported server-side, custom assumes OpenAI-compat or a permissive vLLM/Ollama backend). Local models are unaffected — null capabilities means 'show everything'. chat-adapter.ts now forwards top_k / presence_penalty to the external proxy only when the active provider's capabilities permit it, so the request body matches what the UI shows. * studio/backend: forward top_k to Anthropic; filter OpenAI model list Two paired changes so the frontend capability map has matching backend behaviour: 1. ExternalProviderClient.stream_chat_completion now accepts top_k and forwards it to the Anthropic Messages body. OpenAI-compat providers (which all reject unknown sampling params) still receive only the fields they document. The proxy route in routes/inference.py passes payload.top_k through, so a UI request with top_k actually reaches Anthropic instead of being silently dropped at the boundary. 2. PROVIDER_REGISTRY['openai'] gains a model_id_allowlist regex that scopes the /models picker to current-gen ids (gpt-5.5 / gpt-5.4 / gpt-5.3 / gpt-4.5 / o3 families). The remote /v1/models listing otherwise returns dozens of historical snapshots, fine-tunes and non-chat models (embeddings, TTS, image, moderation) that we never want in the chat UI. default_models is refreshed to match. * studio/chat: relax presence_penalty to optional on OpenAIChatCompletionsRequest Followup to 1fbf445a — chat-adapter now omits presence_penalty for providers that do not accept it (Anthropic / DeepSeek), but the request type still required it as a non-optional number, breaking tsc. The backend pydantic model already defaults presence_penalty to 0, so making it optional client-side matches reality. * studio/backend: route OpenAI traffic through /v1/responses OpenAI's new flagship models (gpt-5.x) return 404 'This is not a chat model' on /v1/chat/completions and are only reachable via /v1/responses. Add a dedicated _stream_openai_responses path in ExternalProviderClient that: - Translates outbound messages into the Responses shape: system messages are folded into the top-level 'instructions' field, user/assistant messages become {role, content} items with input_text / input_image content parts (data URLs and https URLs both pass through). - Drops presence_penalty / top_k / frequency_penalty, none of which the Responses contract accepts. - Translates inbound SSE events back into OpenAI Chat Completions chunks so the frontend keeps a single SSE shape: response.output_text.delta -> delta chunk with content response.completed -> chunk with finish_reason='stop' response.incomplete -> chunk with finish_reason='length' response.failed / error -> propagated error SSE line Stream terminates with data: [DONE] (Responses emits this verbatim). stream_chat_completion dispatches all provider_type='openai' calls to this path; other OpenAI-compatible providers (mistral, gemini, etc.) continue to use /v1/chat/completions. Frontend provider-capabilities map updated to hide presence_penalty for OpenAI in the chat settings panel, matching the new request contract. Includes unit coverage in tests/test_openai_responses_translation.py exercising the request body translation, image-part rewriting, and SSE-to-chat-completions translation via httpx.MockTransport. * studio/chat: clamp external max_tokens to 32k to stay within provider caps The chat settings slider already capped maxTokens at 32768 for external models, but a value persisted from a prior local-model session (where the cap can be 128k+) was sent verbatim to the provider — Claude Opus returns 'max_tokens: 131072 > 128000' on requests like that, and other providers have stricter limits still. Expose EXTERNAL_MAX_OUTPUT_TOKENS from provider-capabilities (32k) and use it both for the slider max and as the clamp inside chat-adapter's external-request body. 32k sits below the tightest declared output limit across the providers we ship and well above what a typical chat reply needs; the local-model path is unaffected. * studio: drop temperature/top_p for OpenAI reasoning models gpt-5.x / o3 / gpt-4.5 are reasoning-class models served via /v1/responses, and reject temperature and top_p with 'Unsupported parameter' 400s. The OpenAI registry allowlist already scopes the picker to those families, so neither knob ever applies on this branch. - external_provider._stream_openai_responses no longer puts temperature or top_p in the request body (kept on the method signature for API symmetry with the other stream methods). - ProviderCapabilities gains temperature/topP flags; OpenAI sets both to false. ChatSettingsPanel hides the sliders for OpenAI so the user does not see inert controls. - chat-adapter omits temperature/top_p from the external request body when the active provider does not advertise them. - OpenAIChatCompletionsRequest type marks both as optional, matching the new chat-adapter shape. - test_responses_request_body_uses_input_and_instructions: assertions flipped to confirm temperature / top_p are absent from the body. * studio: stop forwarding top_k to Anthropic Claude 4.x (Opus / Sonnet / Haiku 4.x) returns 400 'top_k is deprecated for this model' on any request that includes top_k. It was always optional on the older 3.x line, so dropping it unconditionally for every Anthropic call is the simplest path — no per-model gate to maintain. - external_provider._stream_anthropic no longer adds top_k to the Messages body (kept on the method signature for API symmetry). - provider-capabilities sets anthropic.topK = false so the chat settings panel hides the Top K slider for Anthropic providers and chat-adapter does not send top_k in the external request. * studio: gate Anthropic top_k drop to Claude 4.7 only Previous commit (b5aa6ffd) dropped top_k for every Anthropic call, but only Claude 4.7 (Opus/Sonnet/Haiku) actually rejects it. 4.6, 4.5, and the 3.x line still accept top_k and use it as documented. Backend: _stream_anthropic matches the model id against ^claude-(opus|sonnet|haiku)-4-7(-|.|$) and only strips top_k when it hits. Every other Claude generation continues to receive the value from the chat settings panel. Frontend: anthropic.topK is restored to true so the Top K slider is visible again — the backend handles the per-model drop, and the 4.7 case is silent (request still succeeds without top_k). * chore: hide dated openai models in provider select * studio/providers: apply model_id_denylist when listing remote models The OpenAI registry entry gained a model_id_denylist regex matching dated snapshot ids (-YYYY-MM-DD) in 048d73bf, but the list-models route was never consulting it, so the snapshots still showed up alongside their canonical ids (gpt-5.5 and gpt-5.5-2026-04-23 both listed). Apply the denylist with .search() right after the allowlist filter so dated entries are dropped before the response is built. * studio/chat: seed registry default_models for remote providers in picker The Anthropic provider runs in remote model-list mode, so the picker started with an empty availableModels until the user clicked 'Load Models'. If that /api/providers/models call fails (e.g. the known transient decryption error during key rotation), the user sees no models at all — claude-haiku-4-5 in particular was missing from the dialog even though it is seeded in the registry. Always pre-populate availableModels with the registry's default_models when a provider type is selected (curated and remote alike), and have loadModels() return the union of defaults + the live /models response so registry-seeded ids are reachable regardless of what the provider's endpoint returns or whether the call succeeds at all. * studio/backend: diagnostic logging on provider key decryption Decryption failures currently log just 'Failed to decrypt API key: Decryption failed', which leaves no way to tell whether the cause is a stale public key in the browser, a corrupted ciphertext, an unexpected exception class, or a server-side keypair rotation. That's the gap the next reproduction needs to close. - key_exchange now publishes a short SHA256 fingerprint of the public key PEM. init_key_pair logs the fingerprint on generation and warns if it is ever called a second time (re-init silently invalidates every browser that cached the previous public key). - decrypt_api_key wraps both the base64 decode and the RSA decrypt in dedicated try/excepts that log exception type, ciphertext byte length (RSA-2048 should be exactly 256), input string length, and the current public-key fingerprint. - GET /api/providers/public-key returns the fingerprint alongside the PEM so the frontend can correlate a future encrypt-time fingerprint against the decrypt-time fingerprint and prove or rule out a keypair rotation as the cause. - The /test and /models route-level decrypt warnings now include the exception class name (alongside the existing message). * studio/providers: hide dated Anthropic snapshots from the model picker Anthropic's /v1/models returns dated snapshot ids (e.g. claude-3-5-sonnet-20241022, claude-3-5-haiku-20241022) alongside the canonical names users actually want to pick. Same intent as the OpenAI denylist added in 048d73bf, just a different date format — Anthropic uses -YYYYMMDD (no dashes) while OpenAI uses -YYYY-MM-DD. - Add model_id_denylist = re.compile(r'-\d{8}$') to the anthropic registry entry. The /api/providers/models route already applies any denylist after fetching, so dated ids drop out automatically. - Strip the dated 3.5 ids from default_models so the seeded picker no longer surfaces them; keep claude-opus-4-7 and the 4.5 family as the curated set. Net effect: the picker shows opus-4-7 / opus-4-5 / sonnet-4-5 / haiku-4-5 only, regardless of whether the remote /models call succeeds or fails. * fix: provider dialog and mistral short list * style: fix provider dialog curated list styling * fix: provider dialog curated model ids placeholder reference * style: rename Providers to Cloud and tighten dialog header spacing * UX: rename Providers to Cloud, remove header shortcut * studio/chat: normalize structured delta.content from reasoning providers Mistral's magistral (and similarly-shaped reasoning models) stream chat-completion deltas where choices[0].delta.content is an array of structured parts rather than a plain string, e.g. [{ type: 'text', text: '...' }, { type: 'thinking', thinking: '...' }] The accumulator did 'cumulativeText += delta', which coerced each part to '[object Object]' and produced output like '[object Object][object Object]...Hey there!'. Add extractDeltaText() to normalize delta.content before append: - string → returned as-is - array of parts → text/output_text parts contribute their .text or .content; thinking/reasoning parts are re-wrapped inline as <think>...</think> so the downstream parseAssistantContent lifts them into a reasoning part the same way it does for providers that emit thinking inline. magistral keeps its thinking panel; no other provider's output shape changes. - unknown shapes → dropped rather than stringified, so a stray field cannot pollute the rendered chat with '[object Object]'. * Studio: restore Cloud icon shortcut in chat header Brings back the header chip that opens Settings -> Cloud (external providers) directly from the chat view. Same button as before the bf24e604 removal: single-mode only, opens useSettingsDialogStore on the 'connections' tab, tooltip 'API providers'. * studio/chat: strip trailing template literal from external provider streams Mistral's magistral occasionally appends a literal '${response}' token after its actual answer — likely a training-format artifact, since it keeps happening with an empty system prompt and only on that model. Apply a tight strip in the chat-adapter SSE accumulator: when the active provider is external, drop a trailing '${...}' template literal (with optional whitespace) from cumulativeText after each chunk. The regex anchors to end-of-string, so mid-stream fragments ('${re') remain untouched and only collapse once the closing brace arrives. Local-model output is unaffected. * studio/providers: scope Kimi picker to kimi-k2.6 / kimi-k2.5 Mirror what the live Kimi docs surface as the current models (https://platform.kimi.ai/docs/models). Everything else the remote /v1/models call returns — moonshot-v1-* legacy ids and dated k2 previews like kimi-k2-0711-preview — is filtered out. - default_models: ['kimi-k2.6', 'kimi-k2.5'] (was four legacy moonshot-v1 ids plus the dated k2 preview) - model_id_allowlist: ^kimi-k2\.[56]$ applied in the /api/providers/models route after the live fetch - doc-link comments point at platform.kimi.ai overview / models / list-models for the next refresh * studio: drop temperature/top_p for Kimi reasoning models Kimi k2.5/k2.6 are reasoning-class. The API locks temperature and top_p to fixed defaults and 400s on any other value with 'invalid temperature: only 1 is allowed for this model'. The frontend capability map already gated these knobs out of the external request body, but the OpenAI-compat path on the backend unconditionally re-adds them from the pydantic ChatCompletionRequest defaults (temperature=0.7 etc), so the gate was bypassed end-to-end. Add a generic body_omit hook on the provider registry that stream_chat_completion consults after building the body, and use it to strip temperature/top_p for Kimi. Frontend provider-capabilities flips kimi.temperature and kimi.topP to false so the sliders are hidden in the chat settings panel as well. * studio/providers: scope Gemini picker to current 3.x + *-latest aliases Google's /v1beta/openai/models returns dozens of historical, experimental, and non-chat ids that we never want in the chat UI. Cap the picker to the current curated set: - gemini-3.1-pro-preview - gemini-3.1-flash-lite - gemini-3-flash-preview - gemini-pro-latest - gemini-flash-latest - gemini-flash-lite-latest Default_models seeded with these, model_id_allowlist applied in the /api/providers/models route to drop anything else the live fetch returns. * studio/providers: switch Hugging Face to remote model listing Per the Inference Providers docs (https://huggingface.co/docs/inference-providers/index), GET https://router.huggingface.co/v1/models returns the full chat-model catalog across all providers, including per-provider metadata. The OpenAI-compatible endpoint we already use for chat completions accepts the same Bearer token, so flipping model_list_mode from 'curated' to 'remote' lets users discover models via the existing list_models() path without any new wiring. - model_list_mode: 'remote' (was 'curated') - default_models refreshed with current popular ids (gpt-oss-120b, DeepSeek-V3, Llama-3.3-70B, Qwen2.5-72B) so the picker still has a sensible seed if /v1/models fails - notes updated to reference the docs page and clarify the endpoint is chat-only * UX: chat cloud icon changed to model select signifier * studio/providers: org allowlist + count cap for HF Inference picker The HF /v1/models response is the full cross-provider catalog (hundreds of ids — community fine-tunes, mirrors, fp8 variants, dated snapshots). Scope the picker to the first-party org repos worth surfacing and cap the post-filter list. - model_id_allowlist matches the org prefixes openai/, deepseek-ai/, google/, meta-llama/, Qwen/, moonshotai/, mistralai/, zai-org/. Anything outside those orgs is dropped. - model_id_limit (new registry field) caps the post-filter list. The list-models route now slices [:limit] after allowlist/denylist; set to 15 for HF Inference. Other providers leave it unset and behave exactly as before. - default_models stays as the seed so the flagship ids users care about (gpt-oss-120b, DeepSeek-V3, Llama-3.3-70B, Qwen2.5-72B) are always reachable regardless of the API's response order. Dedup is already handled in loadModels() via Set, so no additional work needed there. * style: adjust cloud icon right margin with rem spacing * Studio: cloud openai reasoning level toggle (#5402) * feat: cloud openai reasoning level toggle * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix: honor enable_thinking=false * fix: prevent local reasoning toggle regressions and align OpenAI effort levels * fix: isolate external OpenAI reasoning toggle state --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Roland Tannous <115670425+rolandtannous@users.noreply.github.com> * fix: clamp reasoning effort * fix: align OpenAI reasoning effort * fix: clear stale GGUF badge state * ui: new badge on cloud setting * fix: separate selected models from cached provider model list * Studio: anthropic effort by model family (#5412) * feat: external thinking control and Anthropic effort mapping * fix: anthropic thinking constraints and 4.6 max effort mapping * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix: harden Anthropic thinking params and effort mapping --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * studio/backend: drop top_p from Anthropic body when thinking is enabled PR 5412 added body['top_p'] = max(0.95, min(top_p, 1.0)) inside the thinking branch of _stream_anthropic, but Anthropic returns 400 on extended/adaptive thinking when both temperature and top_p are set: invalid_request_error: temperature and top_p cannot both be specified for this model. Please use only one. (Observed on Claude Opus 4.6.) The contract for thinking-enabled requests is temperature=1 with neither top_p nor top_k allowed. Replace the body['top_p'] = ... line with body.pop('top_p', None). Defensive pop rather than a bare delete: the base body construction above does not currently set top_p, but a future edit that adds it would silently reintroduce the regression. * studio/chat: force reasoningEnabled=true on local reasoning-effort models Followup to PR 5402 / 5412. The model-status refresh path in use-chat-model-runtime carried reasoningEnabled forward verbatim for every reasoning-capable model. That left one observable edge case: 1. user picks an external model that supports Off (gpt-5.x, Claude 4.x), clicks Off — store sets reasoningEnabled=false 2. user switches back to a local reasoning-effort model (gpt-oss / Harmony-style) which does NOT support Off 3. composer's effectiveReasoningEnabled override paints the UI as 'Think: <level>' (on) 4. chat-adapter sees reasoningEnabled=false on the local branch and sends '{}', so the backend's _request_reasoning_kwargs returns None and the Harmony template falls back to its own default effort instead of the displayed level Mirror the composer's override in the store on load: for local reasoning-effort models (where supportsReasoningOff is false), force reasoningEnabled=true so the store and the UI agree on every send. Other reasoning styles still inherit prior state — only the reasoning-effort family changes. * studio/backend: align Anthropic thinking with the extended-thinking docs Two compliance fixes against https://platform.claude.com/docs/en/build-with-claude/extended-thinking 1. Adaptive-mode effort field shape The docs spell adaptive thinking as: {'thinking': {'type': 'adaptive'}, 'effort': {'type': '<level>'}} We had been sending the legacy 'output_config: {effort: <level>}' shape, which Anthropic appears to silently ignore — adaptive ran at the server default effort regardless of the user's selection. Rename to 'effort: {type: <level>}'. 2. thinking_delta event translation The Messages-API streams reasoning content as content_block_delta events with delta.type == 'thinking_delta', which our SSE loop was dropping entirely. On Claude 4.5/4.6 with display=summarized (the default), the user would see the answer text but never the reasoning panel. Wrap thinking_delta.thinking as inline <think>...</think> chunks (same pattern as the OpenAI Responses path) so the frontend's parseAssistantContent lifts it into the reasoning channel. The </think> closer fires on the first text_delta transition, on content_block_stop for the thinking block, on message_delta, and on message_stop — whichever arrives first — so no model path can leak an unclosed <think> into chat output. signature_delta events are left as no-ops; they carry verification metadata, not user-visible content. Adds test_anthropic_thinking_translation.py with httpx.MockTransport coverage of: effort shape on adaptive (Claude 4.6), budget_tokens shape on manual (Claude 4.5), thinking_delta wrapping with signature suppression, and thinking-only turns (display=omitted on Opus 4.7). * studio/backend: revert Anthropic adaptive effort to output_config nesting The previous commit (0a664df4) moved the adaptive-thinking effort field to a top-level 'effort: {type: <level>}' based on a misread of the docs page. The actual Messages API schema nests it under output_config: thinking: optional ThinkingConfigParam ({type: 'adaptive'}) output_config: optional OutputConfig effort: optional 'low' | 'medium' | 'high' | 'xhigh' | 'max' Sending the top-level field produced: 400 invalid_request_error: effort: Extra inputs are not permitted Restore the body to: body['thinking'] = {'type': 'adaptive'} body['output_config'] = {'effort': effort} This was the shape PR 5412 originally shipped (and the author validated against live APIs). My 'compliance fix' was a regression. The companion thinking_delta SSE translation added in 0a664df4 stays — that part WAS missing from the previous shape and is unchanged by this revert. Test pinning the body shape flipped to assert output_config.effort, top-level effort is asserted absent. * studio/backend: opt in to summarized thinking display on adaptive Per the adaptive-thinking docs, the 'display' field on the thinking config defaults to 'omitted' on Claude Opus 4.7 (and Mythos Preview). With 'omitted' the API still emits a thinking content block, but its 'thinking' field is empty — only the signature_delta arrives. Our SSE handler would then surface a stray '<think></think>' for the empty block and the reasoning panel would stay blank for the entire response. Set 'display': 'summarized' explicitly on the adaptive thinking config so Opus 4.7 emits thinking_delta events the same way Opus 4.6 / Sonnet 4.6 do (where 'summarized' is the default, making the explicit setting a no-op there). The manual-thinking branch (Claude 4.5) is unaffected — its default is also 'summarized', and we have no reason to override it. * studio/backend: log Anthropic SSE event counts for thinking diagnostics Reports of 'no reasoning panel content on Anthropic' have two distinct causes that produce the same symptom: 1. Anthropic streamed thinking_delta events but our frontend dropped them somewhere on the rendering side. 2. Anthropic did not emit thinking_delta at all (adaptive mode can skip thinking for simple prompts even with effort=high, and display=summarized only re-enables the *content* — it does not force thinking to happen). Tally each event type for the duration of one stream and log the counts in the finally branch, so the next 'no reasoning content' report shows immediately whether thinking_delta was even on the wire. Zero counts → upstream (model/effort/prompt choice). Non-zero counts → triage moves to chat-adapter / parse-assistant -content / the reasoning component. * studio/backend: route external_provider logs through structlog The studio backend wires structlog as the active logger (via LogConfig.setup_logging at main.py:262), but external_provider.py was using stdlib logging.getLogger(__name__) for every diagnostic. The stdlib root logger defaults to WARNING with no handlers attached, so plain logger.info('...') and logger.debug('...') from this module were being silently dropped — including the 'Proxying chat completion to <url>' and the new 'Anthropic stream event counts' lines. Only WARNING/ERROR survived (via the implicit fallthrough that the user actually observed when an Anthropic call 400'd). Switch the module-level logger to structlog.get_logger(__name__), matching the routes/providers.py and routes/inference.py pattern. All existing call sites use printf-style positional args, which structlog accepts unchanged — no other edits needed. * studio/backend: disable read timeout on SSE streams to external providers Anthropic Opus 4.7 (adaptive thinking) and OpenAI gpt-5.x (/v1/responses) can pause for tens of seconds between bytes while the model is internally reasoning. httpx's read timeout is the *gap* between successive reads, not a wall clock on the whole request — so the shared 120s default was cutting streams mid-response: log: Anthropic stream event counts (... text_delta: 11) Read timeout from anthropic (eleven text deltas in, no content_block_stop, no message_stop) Add a separate _stream_timeout on ExternalProviderClient with read = None (no gap timeout) and the same 10s / 120s connect/write/ pool bounds, then use it at the three SSE streaming call sites: default OpenAI-compat chat completions, _stream_anthropic, and _stream_openai_responses. Non-streaming call sites (chat_completion, list_models, verify_models_endpoint_lightweight) keep self._timeout because a stuck non-streaming response should still fail fast. * studio/backend: log outbound Anthropic request shape for thinking debug After bumping to Xhigh effort the user still saw zero thinking_delta events and only one content_block_start, meaning Anthropic Opus 4.7 opened no thinking block at all. Per the effort docs that should be impossible — Xhigh always thinks. Two open hypotheses: 1. Our adaptive branch is not wiring output_config.effort onto the outbound body for this code path (regex miss, frontend never propagated reasoning_effort, etc). 2. Anthropic is silently accepting output_config as an unknown field and falling back to high default effort regardless. Add a single-line structlog INFO right before the stream POST that echoes the keys actually present on the body (thinking, output_config, temperature, presence of top_p / top_k, max_tokens). Messages are deliberately excluded to keep PII out of the log. With this in place the next 'no thinking on 4.7 at Xhigh' report shows immediately whether we sent the effort knob — separating client bug from provider behaviour. * studio/chat: surface delta.reasoning_content from Kimi / DeepSeek thinking Kimi (kimi-k2.6, kimi-k2-thinking) and DeepSeek's reasoner stream their thinking content via a separate top-level field on the chat-completion delta — choices[0].delta.reasoning_content — rather than as a structured part inside delta.content. Per Kimi docs: In streaming output (stream=True), the reasoning_content field will always appear before the content field. Our chat-adapter SSE loop only read delta.content (via extractDeltaText), so the entire reasoning channel from these providers was being silently dropped — kimi-k2.6 thinks by default yet the chat UI showed no reasoning panel. In the adapter: - Read both delta.content and delta.reasoning_content per chunk - When reasoning_content arrives, open a <think> block in cumulativeText (mirrors how the backend wraps Anthropic thinking_delta and OpenAI Responses reasoning summaries) - When content arrives after reasoning, close </think> first - On stream end, force-close any still-open <think> so parseAssistantContent can lift it into a reasoning part cleanly Anthropic and OpenAI Responses paths are unaffected — they already wrap as <think> on the backend and never set reasoning_content. * studio: Kimi thinking toggle + 16k max_tokens floor Two coordinated changes so Kimi's thinking is user-controllable and the response budget meets the docs' floor. Toggle (frontend + backend): - getExternalReasoningCapabilities now handles provider=='kimi': kimi-k2.6 -> reasoning_style=enable_thinking, reasoningOff allowed kimi-k2-thinking -> always on (reasoningAlwaysOn=true, no off) kimi-k2.5 (and anything else) -> no reasoning controls - chat-adapter already forwards enable_thinking on the enable_thinking-style branch, so the user toggle reaches the backend without additional wiring there. - external_provider stream_chat_completion now translates the boolean into Kimi's wire shape on the default OAI-compat path: enable_thinking=True -> body['thinking'] = {type: enabled, keep: all} enable_thinking=False -> body['thinking'] = {type: disabled} kimi-k2-thinking ignores the toggle so the API never gets a disabled value it would reject. Other providers on the same path are unaffected (gated on provider_type == 'kimi'). Max tokens floor: - New EXTERNAL_MIN_OUTPUT_TOKENS_BY_PROVIDER table and getExternalMinOutputTokens helper. Kimi entry = 16000 per docs: 'Set max_tokens >= 16,000 to ensure the full reasoning_content and final content can be returned without truncation.' - chat-adapter clamps the outbound max_tokens to min(max(stored, providerMin), EXTERNAL_MAX_OUTPUT_TOKENS), so a stored value of 4096 still becomes 16000 when sending to Kimi (other providers unaffected, min stays effectively 64). - chat-settings-sheet's Max Tokens slider min mirrors the same floor when an external Kimi model is selected, so the slider cannot show a value lower than what we'd actually send. - chat-page threads activeExternalProviderType down to the panel. * fix: stabilize external reasoning controls for Anthropic 4.6 and OpenAI o3 normalize Anthropic 4.6 reasoning effort handling by accepting max as an alias and mapping it to xhigh, while keeping Sonnet/Opus 4.6 in default model suggestions. broaden reasoning effort typing across backend/frontend and migrate persisted max selections to xhigh for compatibility. remove reasoning.summary=\"auto\" from OpenAI /v1/responses payloads to avoid o3 eligibility/gating errors. tighten provider model filtering to hide retired gpt-5.3 IDs and add exact/prefix filtering support in provider routes. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * studio: add openrouter/free + full reasoning passthrough on OpenRouter Four-layer wire-up so the OpenRouter free-router model (which picks a free model at random per request, filtered by needed capabilities) shows up in the picker and its reasoning channel surfaces in the chat UI. Registry: - providers.py: openrouter/free seeded at the top of openrouter default_models. Curated list, so picker shows it immediately. Frontend capability map: - provider-capabilities.ts: getExternalReasoningCapabilities now treats openrouter as enable_thinking style with off support. The Think dropdown appears for every OpenRouter model; the gateway silently no-ops the parameter for models that do not reason, so surfacing one toggle on every model is safe. Backend reasoning passthrough: - external_provider.py stream_chat_completion (default OAI-compat branch): for provider_type=='openrouter', translate the request: reasoning_effort in {low,medium,high} -> body['reasoning'] = {'effort': <level>} enable_thinking=True -> body['reasoning'] = {'enabled': True} enable_thinking=False -> body['reasoning'] = {'enabled': False} Matches the documented shape at https://openrouter.ai/docs/guides/best-practices/reasoning-tokens with effort and max_tokens mutually exclusive. Frontend SSE reader: - chat-adapter.ts: OpenRouter streams reasoning as a third shape we did not handle yet: delta.reasoning_details is an array of parts like {type: 'reasoning.text', text: '...'}. Pull text from every part, merge with the existing delta.reasoning_content channel used by Kimi/DeepSeek, and feed the combined string through the same <think>...</think> wrap path so parseAssistantContent lifts it into the reasoning panel. Anthropic/OpenAI Responses paths already wrap on the backend, so they never set this field — no cross-provider interference. * studio/backend: surface OpenRouter SSE errors and router-chosen model in logs The frontend showed 'Provider returned error' for some openrouter/free requests with nothing on the backend side to triage from — the existing 4xx error log only fires when the upstream returns a non-200 status code, but OpenRouter (and most OAI-compat providers) return 200 OK and emit the actual failure as an SSE error event mid-stream, which our default-path stream loop forwarded verbatim without logging. Best-effort diagnostics on the default OpenAI-compat stream path: - Peek at every `data:` line in the inner forward loop, parse JSON best-effort (silently skip on failure so nothing is dropped). - Count event types: delta / error / done. - On any chunk containing an `error` field, emit a structlog WARNING with the provider type and the error payload — same trail the user would otherwise have to dig out of browser devtools. - Latch the first non-empty `chunk.model` field. OpenRouter reports the router-picked underlying model there per request, so the finally-block summary log shows which free model handled the call. In the finally block: 'openrouter stream complete (model=openrouter/free, chosen=google/gemini-2.5-flash, events={delta: 47, done: 1})' Zero overhead for non-error streams (a json.loads per chunk + dict-key lookups). The structlog logger is already configured at INFO; ERROR and WARNING surface in JSON logs without further setup. Hoists `import json as _json` to module top so the default path can reuse it; the existing in-function imports in _stream_anthropic and _stream_openai_responses are now redundant but harmless. * studio/chat: show router-picked model after 'openrouter/free:' in chip When the user picks openrouter/free, the gateway routes each request to a different underlying free model. Until now there was no way to tell which one actually replied without reading the backend logs. Surface the picked model in the active-model chip: - chat-runtime-store gains lastOpenRouterChosenModel: string|null plus a setter. Reset on every model switch unless the user stays on openrouter/free. - chat-adapter SSE loop latches chunk.model into the store on every chunk whose top-level model differs from openrouter/free, gated on the active checkpoint being openrouter/free under an OpenRouter provider. - chat-page externalModels useMemo appends :<chosen> to the display name for the openrouter/free option when the store has a value, so ModelSelector renders e.g. 'openrouter/free:google/gemini-2.5-flash' in the chip. Other models unaffected. - Model-switch callback in chat-page clears the cached value when the user moves to any model other than openrouter/free, so the chip never shows a stale suffix from a previous session. * studio/chat: shorten openrouter/free chip to openrouter:<short-chosen> The full display name in use was: openrouter/free:inclusionai/ring-2.6-1t-20260508:free The `:free` suffix on the underlying id already conveys 'free model', which made the leading `/free` on the router id redundant, and the `inclusionai/` org prefix was just noise crowding the chip. Trim both. Now the chip renders as: openrouter:ring-2.6-1t-20260508:free Strictly a display change in chat-page externalModels useMemo — the backend wire id stays `openrouter/free`, the runtime store still caches the full `inclusionai/...:free` value, and the model-switch clearing logic is unchanged. * studio/providers: switch OpenRouter to remote listing with org allowlist + cap Same shape as Hugging Face Inference. The curated list had only four entries; remote listing fetches OpenRouter's full ~300-model catalog via /v1/models and the new allowlist + limit scope it back down to a usable picker. - model_list_mode: remote (was curated) - model_id_allowlist matches the prefixes: openrouter | openai | anthropic | google | meta-llama | qwen | mistralai | deepseek | moonshotai | inclusionai | zai-org | z-ai Anything outside drops out. - model_id_limit: 20 — first 20 post-filter matches from the live fetch; default_models stays seeded so the most useful canonical ids are always visible regardless of API response order. - default_models seed extended from 4 to 6 (openrouter/free, openai/gpt-4o, anthropic/claude-sonnet-4-5, google/gemini-2.5-flash, mistralai/mistral-large-2411, deepseek/deepseek-r1). openrouter/free remains the first entry, so the dialog's loadModels() union-merge (registryDefaults first, then remote, deduped via Set) keeps it at the top of the picker. * feat: external mistral thinking toggle * studio/chat: fix TS2540 by replacing readonly ContentPart instead of mutating The ContentPart type from @assistant-ui/react marks `text` as readonly, so the coalesce-adjacent-same-type-part optimization in parseAssistantContent failed the tsc build with: parse-assistant-content.ts(15,10): error TS2540: Cannot assign to 'text' because it is a read-only property. parse-assistant-content.ts(25,10): error TS2540: ... This broke npm run build, the Studio installer's `building frontend...` step, and every downstream CI job that runs against an installed Studio (Mac/Windows/Linux variants of Studio API CI, GGUF CI, UI CI, Tauri CI, Wheel CI). Replace the last element with a fresh merged object instead of mutating its `text` field. Same allocation profile as the previous path (one object swap per merge), type-safe under the readonly declaration. Behaviour unchanged. * studio/backend: restore summary='auto' on OpenAI Responses reasoning body A recent refactor dropped the `summary: 'auto'` field from the reasoning config we send to /v1/responses. Without it OpenAI does not emit reasoning summary events on most reasoning models, which means our SSE handler has no <think>…</think> to wrap and the chat reasoning panel stays blank for any gpt-5.x / o3 response. The expected wire shape is: body['reasoning'] = {'effort': '<level>', 'summary': 'auto'} Two backend tests pin this: - test_responses_reasoning_effort_included_when_requested (high) - test_responses_reasoning_effort_xhigh_passthrough (xhigh) Both were failing with AssertionError because the produced body omitted `summary: auto`. Restore the field. Skip it only for the explicit "off" case (effort: 'none'), where summaries serve no purpose. The enable_thinking=True fallback (no explicit effort) also pairs medium effort with summary='auto' so that branch produces reasoning text too. * chat: external reasoning, OpenRouter curation, Think toggle fixes * fix: opus and sonnet 4.6 xhigh --> max * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com> Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com> Co-authored-by: imagineer99 <samleejackson0@gmail.com> Co-authored-by: Daniel Han <danielhanchen@gmail.com>	2026-05-14 16:13:59 +04:00
Lee Jackson	b65a7450ca	Studio: Dark theme refactor, right sidebar redesign, and chat UI polish (#5150) ... * Dark theme refactor, right sidebar redesign, and chat UI polish - Dark theme refactor - Redesign right sidebar - Further left sidebar adjustments - Wider chat and content area; layout tweaks for chat content - Rounded corners across elements for consistency - Show chat message menu icons on menu-area hover, not only on message hover - Assistant message menu icons now always visible; user messages keep on-hover - Redesigned copy icon used consistently across chat blocks and messages - Redesigned trash icon, applied consistently - Unified icon sizing and style with the sidebar - Adjusted icon colors across chat - Fix on-hover background design for chat icons - Fix tooltip from 'more' button staying visible after clicking elsewhere - Adjust position and design of generation speed info text below messages - Adjust design of token speed info popup - Adjust sidebar scrollbar to cover recent chats only * Recents sidebar rename, UI/theme refactor, layout and chat polish UI & Theme: - Dark theme refactor - Consistent rounded corners across elements - CSS polish and cleanup - Remove unused logo image assets Recents sidebar: - Add 'more' button for options menu - Support renaming conversations and training runs - Confirmation dialog before deleting chats - Add optional display_name column to training_runs (idempotent ALTER TABLE) so renaming doesn't lose model_name/dataset_name from the run config - New PATCH /api/train/runs/{run_id} endpoint accepts { display_name: string | null }; empty/whitespace clears the override - Sidebar shows display_name ?? model_name and exposes Rename in the row's More menu, mirroring the chat rename flow - Cache last list response in localStorage and hydrate from it on mount, so recents paint instantly on F5 / route revisit; cached items are shape-validated and dropped if malformed - Optimistic updates on rename and delete (apply locally + cache before background refresh) - Visible toast on rename/delete failure instead of swallowed errors Layout: - Redesigned right sidebar - Further left sidebar adjustments - Updated chat content layout; chat and content area slightly widened - Sidebar scrollbar covers recent chats only Icons: - Redesigned copy icon, unified across chat blocks and messages - Redesigned trash icon to match - Consistent icon sizing and style across chat and sidebar - Adjusted icon colors across chat - Fix icon on-hover background design Chat messages: - Menu icons now appear on hover over the menu area, not just the message - Assistant message menu icons always visible; user messages keep on-hover (next/previous response stays visible for edited prompts) - Repositioned and restyled generation speed info text below messages - Restyled token generation speed popup Tooltips: - Removed tooltip on hover for previous/next assistant response icons - Unified tooltip design across sidebars and chat - Removed tooltip animations (also fixes related lag) Model & Chat Template config: - Merged Chat Template config into Model Configuration section - Added revert-to-original for chat template - Fix Chat Template config disappearing on page refresh until model reload Performance & scroll: - Removed chatbox movement animations across pages/navigation (fixes related UI lag) - Fix scroll flicker at end of streaming when a code block is the final element - Additional chat scroll improvements Bug fixes: - Fix 'more' button tooltip remaining visible after clicking elsewhere * Remove sidebar localStorage cache and optimistic updates Drops the localStorage hydration and optimistic rename/delete logic from the recents sidebar; reverts to fetching fresh on mount. * Fix missing cn import in shared-composer (regression from merge) * chore(sidebar): import sidebar deps from feature indexes Re-export deleteChatItem / renameChatItem / useChatSidebarItems / SidebarItem / useChatSearchStore / ChatSearchDialog from @/features/chat, and removeTrainingUnloadGuard from @/features/training. Switch app-sidebar.tsx to consume them via the public feature indexes instead of deep paths, clearing the no-restricted-imports eslint errors. No behavior or UX change. * fix(studio/frontend): reload training Recents sidebar after F5 refresh The Recents sidebar showed empty after a hard refresh. The hook's inFlightRef dedup guard collided with React StrictMode's double-mount in dev: the second mount's fetch returned silently with no error, no retry, and no toast — leaving the sidebar empty until navigation. Replace skip-if-busy dedup with abort-previous via a hook-level AbortController. This also fixes a latent race where a slow poll could resurrect a just-deleted row by clobbering the optimistic update. Changes (all in use-training-history-sidebar.ts): - fetchRuns aborts any in-flight request before starting a new one; post-await signal.aborted check drops stale responses. - Optimistic helpers (applyRunUpdate, removeRun) abort in-flight fetches so they don't depend on caller discipline to invalidate stale data. - Initial load gets bounded retry-with-backoff (500ms / 1.5s / 3.5s) and surfaces a sonner toast with a Retry action on final failure. - Failure toast auto-dismisses on any successful load (initial retry, Retry click, or polling recovery). - Polling pauses while the tab is hidden and catches up on visible, avoiding wasted requests during long training runs. - Both effects own their teardown explicitly (abort + clear timer). * Apply unified tooltip design and behavior across remaining pages for consistency * UI polish: spacing, tooltip on source icons, letter spacing, smaller icons, consistent edit icon - Adjust tiny spacing between elements around the UI for subtle polish - Redesign tooltip on source icons for web search / tool use, consistent with the new design - Adjust chat text letter spacing - Smaller icon sizes - Replace 'edit message' icon in chat with the new Rename icon used in Recents for consistency * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Adjust CSS for right sidebar * Fix scrollbar UI compatibility across browsers * fix: preserve chat preset settings on model load * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix(studio): remove duplicate chat template status field * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * chore: remove creative preset assumption * fix(studio): align speculative decoding default * fix(studio/chat): snap numeric param inputs to step grid - Type a value in any param input (Temperature, Top K, Max Tokens, etc.) now clamps to [min, max] and snaps to the slider's step grid, killing off-grid values like 1.051234 and FP residue from slider drags. - Branch picker chevrons share the action bar's 32px height + 10px radius via a new .aui-branch-chevron-btn utility; hover area aligns visually while staying narrower than the sibling icon buttons. * fix(studio/chat): keep training-run polls converging and drop dead preset code - Keep training-run polls converging when responses outrun the 5s interval (don't unconditionally abort prior in-flight; skip if one is still pending, mutation race still guarded). - Drop dead Creative/Precise preset code paths (remove 'builtin-fixed' source variant + unreachable branches). * fix(studio): training-run cards show custom name + model + dataset - Training-run cards now display custom display_name + model + dataset, with cross-view sync on rename/delete. - Enhance clarity of borders and colors in dark theme on export etc. * fix(studio): match active state green to unsloth brand color * fix(studio): preserve can_resume on training rename * fix(studio): keep GGUF chat template override distinct * fix(studio): treat audio input models as multimodal * fix(studio): cancel numeric draft on Escape * fix(studio): use default speculative mode on toggle * fix(studio): detect GGUF audio VLM input models * fix(studio): address final PR review findings * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix(studio): refresh sidebar/history when a new training run starts so it appears without a manual reload * fix: API and svg * fix(studio/sidebar): align run rename dirty check with displayed baseline * fix(studio/sidebar): use leading-tight on account block to prevent descender clipping with truncate --------- Co-authored-by: sneakr <hauzin@hotmail.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Roland Tannous <115670425+rolandtannous@users.noreply.github.com> Co-authored-by: shine1i <wasimysdev@gmail.com>	2026-05-07 14:33:31 +04:00
Lee Jackson	2de17c0a96	Studio: Add checkpoint resume for stopped training runs (#5255) ... * feat: add checkpoint resume for stopped training runs * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix:add resume checkpoint helpers * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix: use checkpoint parent as resume output dir * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix: save optimizer and scheduler state on stop-and-save Use Trainer._save_checkpoint instead of save_state so resume restores optimizer momentum and LR-schedule position via the checkpoint-NNN/ subdir written by HF's official path. * fix: clean up resume training history and startup progress * fix: preserve resume output dirs * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix: tighten resume run lookup * fix: remove stale output-dir lookup * fix: preserve startup download progress --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Roland Tannous <rolandtannous@gravityq.ai> Co-authored-by: Roland Tannous <115670425+rolandtannous@users.noreply.github.com>	2026-05-04 00:34:46 +04:00
Wasim Yousef Said	1e8875584d	feat: custom scan folders for GGUF model discovery (#4723) ... * feat: add scan_folders table and CRUD functions to studio_db * feat: add scan folders API endpoints and integrate into model scan * feat: add scan folders API client and update source types * feat: add custom source to model filters and selector * feat: add Model Folders section to chat settings sidebar * style: fix biome formatting in ModelFoldersSection * fix: address review findings for custom scan folders empty string bypass, concurrent delete crash guard, Windows case normalization, response_model on endpoints, logging, deduplicated filter/map, module level cache for custom folder models, consistent source labels, handleRemove error surfacing, per folder scan cap * fix: show custom folders section regardless of chatOnly mode * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * refactor: extract shared refreshLocalModelsList in pickers * Harden custom scan folder validation and scanning - Validate path exists, is a directory, and is readable before persisting - Apply per-folder model cap during traversal instead of after (avoids scanning millions of inodes in large directories) - Wrap per-folder scan in try/except so one unreadable folder does not break the entire /api/models/local endpoint for all callers - Normalize case on Windows before storing so C:\Models and c:\models dedup correctly - Extend macOS denylist to cover /private/etc and /private/tmp (realpath resolves /etc -> /private/etc, bypassing the original denylist) - Add /boot and /run to Linux denylist * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Improve scan robustness and preserve Windows path casing - Preserve original Windows path casing in DB instead of lowercasing (normcase used only for dedup comparison, not storage) - Catch PermissionError per child directory so one unreadable subdirectory does not skip the entire custom folder scan - Wrap list_scan_folders() DB call in try/except so a DB issue does not break the entire /api/models/local endpoint * fix: scan custom folders for both flat and HF cache layouts * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix Windows case-insensitive path dedup with COLLATE NOCASE Use COLLATE NOCASE on the scan_folders.path column so that the UNIQUE constraint correctly deduplicates C:\Models and c:\models on Windows without lowercasing the stored path. Also use COLLATE NOCASE in the pre-insert lookup query on Windows to catch existing rows with different casing. * Restore early-exit limit in _scan_models_dir for custom folders Keep the limit parameter so _scan_models_dir stops iterating once enough models are found, avoiding unbounded traversal of large directories. The post-traversal slice is still applied after combining with _scan_hf_cache results. * feat: scan custom folders with LM Studio layout too * Fix custom folder models being hidden by dedup Custom folder entries were appended after HF cache and models_dir entries. The dedup loop kept the first occurrence of each model id, so custom models with the same id as an existing HF cache entry were silently dropped -- they never appeared in the "Custom Folders" UI section. Use a separate dedup key for custom-source entries so they always survive deduplication. This way a model can appear under both "Downloaded" (from HF cache) and "Custom Folders" (from the user-registered directory) at the same time. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Harden LM Studio scan and fix COLLATE NOCASE on Linux - Add per-child and per-publisher OSError handling in _scan_lmstudio_dir so one unreadable subdirectory does not discard the entire custom folder's results - Only apply COLLATE NOCASE on the scan_folders schema on Windows where paths are case-insensitive; keep default BINARY collation on Linux and macOS where /Models and /models are distinct directories * Use COLLATE NOCASE in post-IntegrityError fallback SELECT on Windows The fallback SELECT after an IntegrityError race now uses the same case-insensitive collation as the pre-insert check, so a concurrent writer that stored the path with different casing does not cause a false "Folder was concurrently removed" error. --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Daniel Han <danielhanchen@gmail.com>	2026-03-31 06:40:31 -07:00