49 commits

Author	SHA1	Message	Date
Daniel Han	78a3587427	Keep path_utils off PEP 604, which the 3.9 floor gate rejects (#9335) ... * Keep path_utils off PEP 604, which the 3.9 floor gate rejects `Core` has been failing on main at 'python floor compatibility (HARD GATE)', all three HF/TRL cells: 36 studio files now evaluate PEP 604 unions on the floor, up from 35 The 36th is studio/backend/utils/paths/path_utils.py, from #8919. That change is right; it just added subject_key: Callable[[str], object] | None = None to a module with no `from __future__ import annotations`, so the annotation is evaluated at import and `X | None` is a TypeError on the declared 3.9 floor. Uses Optional[...], which the module already imports, rather than adding the future import. Five other modules import this one, and deferring every annotation in it is a wider change than the one union needs -- the ratchet's own docstring warns that converting a file wants Studio booted, because FastAPI resolves annotations when it builds endpoints. Verified on main: offenders back to 35 against a debt of 35, the whole test_python39_compatibility.py suite passes (8), and the AppleDouble guards the original change added still pass (9). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-08-19 20:58:36 -07:00
sts-change	8d6e969378	Studio: never pick a macOS AppleDouble sidecar as a GGUF (#8919) ... --------- Co-authored-by: Lyxot <longyixing331@gmail.com> Co-authored-by: oobabooga <112222186+oobabooga@users.noreply.github.com>	2026-08-19 15:51:16 -03:00
Michael Han	0eb6b6c931	Studio: say when a scan folder cannot be read instead of showing no models (#9053) ... * Studio: say when a scan folder cannot be read A folder Unsloth is denied looks exactly like an empty one: the scan catches the OSError, logs it, and moves on, so the model list is empty with no reason given. Add-time validation now opens the directory instead of trusting os.access, which reads mode bits only and passes on folders macOS TCC or a Windows ACL still refuses. The check runs after the denylist rules so a denied path is never opened. The scan keeps the error it already caught, and both scan-folders endpoints return it as a per-folder status the dialog shows with the setting that fixes it. No new work on the healthy path: 0.06us per folder per scan, and the folder list is a dict lookup with no syscalls. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: record scan folder status from the Hub inventory scan too The folders dialog reads /api/hub/scan-folders, but the scan behind it is the Hub inventory, not collect_local_models, so nothing was ever recorded for it and every row stayed "ok". Its custom-folder loop now records the same way. That alone is not enough: the Hub's _scan_models_dir catches the OSError itself and returns an empty list, so no exception reaches the loop. So an empty result is now the trigger. One opendir says whether the folder is empty, gone, or refused, and it runs only for a folder that returned no models. A folder that found models still costs nothing, with a test that fails if it ever touches the filesystem. * Studio: catch denied model subdirs, and recheck when the dialog reopens Two gaps in the folder status. A root can list fine while every model under it is denied, on a NAS mount or a drive owned by another user. The scanners skip an unreadable child silently, so that arrives as the same empty list as an empty folder and was reported as ok. The probe now also opens subdirectories, stopping at the first refusal and capped at 64, so the denied-everything case costs one extra open. The row tells the user to fix permissions and reopen the dialog, but nothing rechecked between inventory scans, so the warning stayed up after access was restored. Listing the folders now rechecks the folders marked bad, and only those. A healthy folder is not in the registry, so the list still opens nothing. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: probe two levels down, and keep the tests collectable on Windows The child probe descended one level, so <root>/<publisher>/<model> with the model denied still reported ok: both levels above it list fine and the scanners return nothing. It now walks two levels, depth first, on one shared budget of 64 opens. Depth first means a denied mount is found in three opens instead of after every publisher, and the budget bounds the cost whatever the shape of the tree. os.geteuid does not exist on Windows and a skipif condition is evaluated at import, so collecting the test file there raised AttributeError before the os.name check could skip anything. Resolved once into a shared marker, with a test that runs the module body with geteuid removed. * Studio: flag a folder with one denied model, and show status in the picker A folder holding one readable model and one denied model returned the readable one, so the scan looked successful and the denied model was silently absent. The probe now runs whether or not models were found, and reports "partial" in that case so the copy does not contradict the rows on screen by claiming the folder cannot be read. That is a real cost change on the healthy path, so it is measured rather than claimed: 104us for a folder with 8 model dirs, 0.77ms for one with 300, capped by the same 64-open budget. The end-to-end scan stays inside run-to-run variation, and the folder list still opens nothing unless a folder is already marked bad. The old zero-syscall test is replaced by the bound, which is now the guarantee that matters. The inline model selector manages the same folders and rendered only the path, so it shows the status too. * Studio: stop treating an exhausted probe budget as healthy Three fixes. A denied directory past the open budget was reported as ok. Running out of budget means the tail was never looked at, which is not the same as finding it healthy, so it now returns an internal "unknown" that is never recorded and never sent to the UI. That alone would strand a wide folder in a warning it could never clear, so the registry now remembers which directory refused. A recheck opens that one directory, which settles a fixed folder in a single open no matter how wide the folder is or where the denial sat. A folder recorded as partial kept that status after being deleted, because the recheck preserved partial for every non-ok probe. It now only holds partial against a permission result, so missing and unreadable replace it. One permission test was missing the marker that skips it as root and on Windows, where chmod 000 does not deny. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Probe scan folders off the event loop, and stop a vanished model or a Windows device error condemning the folder * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Probe the commit directory inside an HF snapshots folder, and stop calling a shut root partial Two ways the dialog told the user the wrong thing. The cache layout is models--org--name/snapshots/<commit>/, three levels under a registered root, so a probe that stops at two never opens the one directory the weights live in: a denied commit dir made the model vanish from the list while the folder still reported ok, which is the silent empty case this PR exists to remove. The extra level is bought only for a directory named snapshots, since a blanket third level would spend the open budget descending into diffusers component directories. And when the root itself becomes denied, the partial branch restored partial regardless, so a folder none of which can be read kept saying some models in it could not be read, sending the user hunting for one bad model. The cause the probe already returns is the discriminator: it is the root path itself for the root's own refusal and a nested entry path otherwise. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Daniel Han <danielhanchen@gmail.com>	2026-08-19 06:10:58 -07:00
Michael Han	eb63760b6f	Studio: find the real Documents folder on Windows for project workspaces (#8955) ... * Studio: find the real Documents folder on Windows for project workspaces Project workspaces are the only thing Studio writes to Documents, so a Documents folder resolved wrong breaks project creation and nothing else. OneDrive's Known Folder Move repoints it and leaves ~/Documents behind. Also report which folder could not be created, instead of a bare 500. Fixes #8936 * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: only blame the projects folder for a projects folder failure The same upsert opens studio.db first, which is a different path. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: stub the workspace refusal instead of staging it with chmod Root ignores a read-only directory and Windows does not enforce one. --------- Co-authored-by: Unsloth <michaelhan@Michaels-MacBook-Pro.local> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-08-15 22:48:24 -07:00
Maheswar Kumar	e2b86860de	add a way to open a chat's sandbox folder from the ui (#8661) ... * add a way to open a chat's sandbox folder from the UI Tool calls write into a per-chat sandbox under the studio home, and the only way to find that folder was to search the filesystem by hand. The path is never shown anywhere in chat. Add POST /api/inference/sandbox/{session_id}/reveal, which resolves the session's sandbox off the event loop and opens it in the OS file manager. It 404s rather than creating a folder for a chat whose tools never ran, and re-resolves once so the legacy sandbox move cannot make an existing folder look missing. _reveal_in_file_manager moves from routes/models.py to utils/paths/path_utils.py so the cached-model reveal and this one share one implementation, keeping the macOS, Windows, WSL and xdg-open branches in a single place. Two entry points in the UI, both desktop-only because the file manager is the backend host's: "Open chat folder" in the sidebar chat menu, and the "files created" heading on python and terminal tool cards. In a browser the menu row explains why it is unavailable, and the heading stays plain text. A compare row outside a project spans two sandboxes, so it gets no entry. Threads inside a project resolve to the project workspace, matching where their tool calls actually run. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * open the sandbox a chat's files are actually in test_reveal_file_manager.py called routes.models._reveal_in_file_manager, which moved to utils/paths/path_utils.py, so all five platform tests failed with AttributeError. They already import path_utils to patch _IS_WSL; point the calls there too. Moving a chat between projects rewrites only its projectId, while the files its tools wrote stay where they were. Deriving the folder from current membership therefore opened the project workspace for a chat whose artifacts are under its thread sandbox, or 404'd when that workspace did not exist. Read the session id back from the chat's own tool results and fall back to the current scope only when it never ran one. * read a recorded sandbox only from this app's own tool wrapper The scan accepted any result object carrying a nonempty sessionId, so a custom or MCP tool answering with one of its own would have been read as sandbox metadata and the sidebar would open an unrelated folder, or 404. Gate it the way the adapter already gates unwrapping: the part must be a tool-call whose name is in SANDBOX_FILE_TOOLS, and the result must match the full wrapper shape rather than merely having text and sessionId. isSandboxFileList and isSandboxToolResult move from chat-adapter.ts into sandbox-files.ts so both callers share one copy instead of the check being restated. That module is pure, so the predicate no longer drags the chat adapter's import graph into a small utility. * refuse to reveal a sandbox that is already gone reveal_in_file_manager falls back to the parent directory on Linux when the target is not a directory, so a sandbox deleted between the route resolving it and the worker opening it would have opened the root holding every other chat's folder. Raise FileNotFoundError instead, which the route reports as the same 404 a missing sandbox already gets. A compare row moved into a project can still hold one folder per pane, since each pane ran under its own thread sandbox before the move. Reading only the first pane opened its folder and silently ignored the second, so read every pane and offer the action only when they agree. Two source-text tests read the sandbox wrapper predicate out of chat-adapter.ts, which it moved out of in the previous commit. Point them at sandbox-files.ts, where the adapter and both tool cards now share it. * Fail closed when a reveal target vanishes after the guard The existence check at the top of reveal_in_file_manager and the is_dir() in the Linux branch are two separate stats, so a sandbox deleted between them still reached "directory = str(path.parent)". The parent of a sandbox is the root holding every other chat's, which is the exact widening the guard was added to prevent. Take the parent only for something positively a regular file, and raise for anything else. No live path changes behaviour. Also completes the platform matrix. macOS and native Windows had no coverage at all, and neither did any of the ways the call can fail once a branch is chosen. The Windows one is worth naming: explorer.exe returns 1 even when it worked, so Popen without a wait is load-bearing, and a run(check = True) there would raise on every successful reveal. * Stop reporting a missing file manager as a missing folder Popen raises FileNotFoundError for the LAUNCHER as readily as for the target, so on a host with no xdg-open (headless, a container, a plain server install) a chat whose files were sitting right there was told it has no folder. That sends the user looking for a bug in the chat instead of installing a file manager. Only answer 404 when the directory really has gone; otherwise take the sanitised 500 that was already there. Adds the route coverage the endpoint was missing: that it authenticates before it touches the filesystem, that it reads the ?session= fallback the frontend uses for ids a path segment cannot carry, that a traversal id derives a name inside the root and launches nothing, that a sandbox file literally called "reveal" is still served because the new route is POST-only, and that the cached-model reveal still goes through the moved helper for the symlinked entries a Hugging Face cache is made of. * Report a failed history read instead of guessing the folder from it The per-pane read caught its own rejection into undefined, which made a failed read indistinguishable from a chat that never ran a tool. The handler then fell back to the current project scope, which is the answer the recorded session id exists to override, so a moved chat could be sent to a folder it never wrote to. Let it reach the report below. The browser row also carried its explanation only in a tooltip hanging off an aria-hidden anchor, opened on hover. A screen reader never reaches it and a touch device has no hover, so both were told the row does nothing and not why. Selecting it now opens the hint, and the row carries the same sentence as a title, which is what the desktop variant already does. Tests: the reveal path helper had no consumer and no coverage, including the case that matters most, an id the router cannot carry landing as /sandbox/_/reveal?session=... rather than putting the query first. Plus the message a caller shows for the backend's own reason, a non-JSON body, and an old backend answering 405 because the download route claims that path for GET/HEAD. The recorded-session scan gains legacy content shapes, null parts and a 5000 message history. * Read the panes one at a time, as this file's export contract requires tests/studio/test_desktop_reliability_frontend_contract.py asserts that `await Promise.all(` appears nowhere in app-sidebar.tsx: every batch in this file ends in a native save dialog, and concurrent ones race each other and lose cancellation. The sandbox reads are not saves, but the guard is file-wide and this is the first `await Promise.all(` the file has ever had, so Repo tests (CPU) fails on it. Sequential instead. A compare row has two panes, so there is nothing to win by overlapping them, and the loop stops on the first failed read rather than carrying an undefined through. Asserted from the frontend side too, so the reason sits next to the code and not only in the Python guard that fails the build. * Record the sandbox on every python and terminal result A run whose result carried no __FILES__ or __IMAGES__ envelope was persisted as a bare string, so nothing in the stored chat said which sandbox it ran in. That is not only the harmless "wrote nothing" case: _created_file_sentinels returns an empty string when a concurrent call shared the directory, which is the normal shape of a project workspace, so a call that DID write files can arrive bare too. Move the chat afterwards and the menu falls back to current membership, opening a folder it never wrote to or 404ing, while the files sit in the old workspace. The session the call ran in is already resolved at that point, so record it. Transparent downstream: toolResultModelText and the outbound translator both unwrap a sandbox wrapper to .text, which is the same string the bare branch stored, including the empty-output sentinel, and the cards already accept both shapes. The "files created" row still hides itself on an empty list. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Refuse to reveal a sandbox that is no longer a directory Deletion was not the only way into the parent. A tool runs inside its own sandbox and can replace it with a regular file, and every platform's file branch then names the enclosing directory, which for a sandbox is the root holding every other chat's: xdg-open on the parent, open -R and explorer /select, on the parent with the file picked out. Re-checking in the route cannot close that, since the swap lands wherever it lands. So the caller states what it is revealing instead: expect_dir refuses anything that is not a directory, and the sandbox route passes it. The type is now decided once and only read afterwards, rather than each branch re-stat'ing and possibly disagreeing with the guard above it. Off by default, because the cached-model reveal legitimately points at a file. Recover a legacy chat's folder before falling back to its current scope Chats stored before results carried a session recorded nothing, so one that ran while it was loose and has since joined a project would be sent to the project workspace instead of the folder it actually wrote. Its own thread sandbox is the only other candidate that can be named, and files sitting in it are this chat's, so it is probed first and the existing "more than one folder" guard still arbitrates a compare row. Only in that direction: a chat moved OUT of a project wrote under project-<id> and nothing retains which project that was, so that case still falls back. The probe costs one listing request, and only for a chat in a project that recorded nothing. * Refuse a sandbox that is a symlink, in the same stat as the type check is_dir() follows links, so a sandbox replaced by a link to somewhere else passed the directory check and the launcher opened the target: another chat's sandbox, or any directory at all. Asking whether the path is a link and then whether it is a directory would be two stats and the same race again, so expect_dir now takes one lstat and reads the link's OWN type. That answers both questions at once and leaves no window between them. is_dir(follow_symlinks = False) would say the same but only on 3.13, and this runs on 3.10. Unchanged without the flag: a Hugging Face snapshot is a link farm, so the cached-model reveal still follows links, and there is a test for each. Report a probe that could not be answered instead of guessing from it A sandbox that does not exist already answers 200 with an empty list, so a non-OK response from the legacy probe is a real failure, not an empty folder. Reading it as "no files" sent the caller on to the current project scope and opened a different workspace while saying nothing, which is the same masking this branch removed from the history read two commits ago. * Shorten the comments added by this PR Same intent, fewer lines: collapse the multi-paragraph docstrings on the reveal helper and its tests, and drop the repeated explanations in the sidebar handler, which said the same thing three times over. Comments and docstrings only, no code touched. --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Daniel Han <danielhanchen@gmail.com>	2026-08-13 17:09:20 -07:00
Daniel Han	ba4f7274a1	Studio: show the files a tool call creates, and keep them in one place (#8256) ... * Studio: show the files a tool call creates, and keep them in one place Files written by the python or terminal tool were unreachable from the chat: only images could be fetched back, nothing could list a chat's files, the terminal tool reported none at all, and the model was never told what happens to what it writes, so it guessed at a temporary directory and users had to search the disk. - both executors report created files via a __FILES__ sentinel, stripped before the model sees it - non-images download as attachments, images still render inline - GET /sandbox/{session_id} lists a chat's files and its path - tool cards get a 'files created' row with working downloads - the sandbox moves under the studio home (migrated, overridable) instead of a third folder in the user's home - UNSLOTH_COMPILE_LOCATION is pinned before unsloth_zoo reads it, so the compiled cache stops landing in the launcher's working directory - deleting a chat cleans up its sandbox * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Do not create a sandbox from a read, and reject reserved names Serving or listing a file resolves the sandbox path instead of creating it, so a GET no longer leaves a folder behind for every id it is asked about. Session ids matching a Windows device name (CON, NUL, COM1) now collapse to _invalid, and the compiled-cache cleanup follows the configured location rather than only the defaults. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Cover the cases the first pass missed Files written into a subdirectory are now found, listed and downloadable, and a run that wrote its file before timing out or being cancelled still reports it. Clearing all chats removes their sandboxes like deleting them one by one does. The legacy migration is serialised and runs for a read too, so an upgraded install does not 404 until some tool happens to run. The image list is capped like the file list, the __FILES__ envelope is only stripped when it parses, and a direct `uvicorn main:app` start pins the compiled-cache location as well. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Replay the tool text, and keep the sentinel honest A file-producing call was replayed to the model as the whole card object rather than the stdout it saw; the same shape was already reaching that path for image results, so both go through their text now. The snapshot walk and the download route share one segment limit, so the card cannot advertise a file the route would refuse, and both ends of the __FILES__ envelope check each entry rather than only that it is a list. Size joins mtime in the change key for volumes with coarse timestamps. * Drop the duplicate import the main merge left behind * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Report exactly what the route will serve The snapshot and the listing apply the download route's own per-segment allowlist, so a name containing a backslash or a control character is never offered as a chip that would then 404. Dotfiles a tool wrote are reported now, since .gitignore is a real artifact and the route serves it; dot-directories stay out, which is where the noise lives. Deleting a chat runs the legacy migration first, so a delete before any tool has run still finds the folder. * Do not delete what is not ours, and do not serve what is not inside UNSLOTH_COMPILE_LOCATION is user-set, and routing the cleanup through it let a value naming a shared directory take that whole tree at startup; a configured path now has to look like a compiled cache before anything deletes from it. The read-only resolver applies the containment check the executing one always had, so a session entry that is a symlink out of the root no longer becomes the root and serves its target. The listing walk drops directory segments the download route would refuse, and executor scratch no longer keeps a deleted chat's folder alive forever. * Unwrap the result everywhere it is read, not just on replay The export paths feed fine-tuning datasets, and they serialised the card wrapper whole; they share one helper with replay now, which also covers the image shape they never unwrapped either. A file the user named studio_exec_results.csv is no longer treated as the executor's scratch, which is specifically studio_exec_<random>.py. And a sandbox root Studio did not create keeps its permissions, since the override can name a shared directory. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Prove the cache is ours before deleting it A directory of plain .py files is somebody's package, so the shape test was still too loose: Studio writes a marker file when it creates the location, and a configured path needs that marker or a generated module name before anything deletes from it. Containment moved into one helper applied to cached session paths too, so a directory swapped for a symlink after it was cached stops resolving there. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Delete only our own files from a shared compile location A directory holding a generated module was treated as ours whole, so a UNSLOTH_COMPILE_LOCATION pointing at a shared directory lost its siblings. Only a directory Studio created (marker) is cleared whole; anywhere else just the generated modules go. Also cap directories visited during the sandbox snapshot, and treat a RecursionError from an oversized __FILES__ payload like the invalid JSON it is. * Bound the listing walk, and retry a legacy move that failed The listing route grew its own copy of the walk and drifted from the snapshot: no directory budget, a prefix match that hid the user's own studio_exec_results.csv, and a hardcoded file cap. It now shares all three with the snapshot walk. Migration flagged itself complete even when a move raised, so a file locked on Windows was stranded once the destination directory appeared. It reports whether anything movable is left, and only then is it done. Clear-all-chats takes the same delete_files opt-in as DELETE /threads, still off by default. * Never delete through a session symlink, and keep the cache marker A session entry replaced by a symlink to a sibling passed the realpath containment check, so deleting that chat took the other one's files. The link itself is dropped and its target left alone. Startup clears the cache it just created, and nothing rewrote the marker afterwards, so the next cleanup demoted our own cache to shared. It is restored after clearing a directory we own. A chat's first turn has no thread id yet but still runs in a real workdir (_default, which the UI and the download route both resolve), so its files get a download chip too. The model- and API-facing message is still stripped, which is the boundary that has to stay clean. * Drop a stray file a local test run left in the tree * Reserve no filename, and prove ownership of a CWD cache The executor's scratch script moves out of the sandbox, so a tool writing studio_exec_results.py keeps it: nothing in the sandbox belongs to Studio except the remap sidecar it writes itself, which is excluded by exact name. That sidecar was being reported as a user file, which also made a streamed result differ from a non-streamed one. A folder named unsloth_compiled_cache in the launch directory is no longer ours by construction; it needs the marker or generated modules like any other configured location. Containment uses commonpath, so a sandbox root that is a filesystem or volume root no longer sends every chat to _invalid. * Put the scratch script back in the sandbox, and gate first-turn files Moving it to the system temp directory changed sys.path[0], so a module an earlier call wrote stopped importing, __file__ pointed outside the sandbox, and the script's directory became world-writable /tmp. It is back in the workdir; the reporting exclusion is now this call's exact filename rather than a pattern reserved over user names. A turn with no chat id runs in the shared _default workdir, so reporting its files pinned a card to a directory the next new chat would overwrite. Gated again until a per-chat sandbox exists. register_compiled_cache_on_path applies the same ownership test as cleanup, so an unrelated folder in the launch directory cannot shadow dependencies. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Do not unlink a sandbox a tool is using, or claim a directory we found Deleting a chat mid-call removed the empty workdir out from under the running tool, and a process whose cwd is gone fails every relative write. Sessions with a call in flight are tracked and refuse removal. mkdir(exist_ok=True) does not mean Studio created the directory, but the marker that followed licensed an rmtree of it. It is written only when this call actually made the directory. * Decide and delete under one lock, and offer chats the same file choice The busy check released the lock before touching the filesystem, so a tool could start in the window and end up running in a directory the same call then removed. The check and the unlink are one critical section. Chat delete now offers the same switch project delete has, so a chat that wrote files is not left with an unreachable folder. Off by default, and the sandbox is only removed when the user asks. * Rename the delete switch state now that chats use it too * Queue a refused delete, and only offer the switch where it works A delete refused because a tool was running was dropped, and the thread was already gone from history, so nothing would ever name that session again. It is queued and performed when the call ends. The delete switch appeared for training runs and for chats inside a project, where nothing it promises happens. It is shown only where a sandbox can actually be removed, and every opener clears it so it cannot arrive preselected from a cancelled dialog. rmtree refuses a symlink and ignore_errors hid it, so a symlinked UNSLOTH_COMPILE_LOCATION was never cleared. Resolved first. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Tighten the comments in the sandbox and cache paths * Prove ownership before following a cache link, and contain the fallbacks Resolving the cache path made rmtree follow a link, so a built-in path that is a symlink would have taken its target with it. Being at a built-in path proves ownership of the directory, not of what a link there points at; only the marker on the target does. _default and _invalid are ordinary directories in a writable sandbox, so one replaced by a symlink became the root every unchecked request read from. They go through the same containment as a session. A file delete now detaches the directory under the lock and removes the tree afterwards, so a large sandbox no longer blocks every tool start or the event loop of the route that asked. * Finish a detached delete a restart interrupted The detached directory sits under a name no session id can reach, so an exit mid-delete stranded it for good, which is the accumulating folder this change is about. Each delete clears the leftovers of earlier ones. * Sweep interrupted deletes at startup, and stop guessing at project ids The delete worker is a daemon, so an exit left the renamed directory with nothing to reclaim it until someone deleted another chat. Startup sweeps them once, off the request path. A session id starting with project- is only a project workspace when the project exists; an imported chat carrying the prefix gets an ordinary directory from _get_workdir and was never cleaned up. Studio's tool wrapper always carries images, so requiring it stops a foreign result with text and sessionId from being reduced to its text on replay and on every export path. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Name the encoding in two test writes * Prove a sandbox is ours before deleting it The root can be an existing shared folder the user pointed us at, so a chat id can name something already in there. Every session directory we create now carries a marker, and outside our own root that marker is what a delete needs. The tombstone sweep matched any name containing .deleting-, which in such a root reached a user's own report.deleting-backup. It now matches only the exact name the rename produces. Two ids differing only in case are one directory on Windows and on a default macOS volume, so the in-flight bookkeeping folds them together while the queued delete still carries the exact id. * Only ever reclaim what Studio itself created The marker was written on every resolve, so a chat id naming a folder that was already in a shared root claimed it. It is written only when the call created the directory, and a tombstone now needs the marker too, so a user's own archive.deleting-<hex> is left alone. Queued deletes are held per exact id under the folded key: on a case-sensitive filesystem two ids that differ only in case are two directories, and the second request used to replace the first. Deletion in a cache directory we do not own is limited to unsloth_compiled_module_*.py. Unsloth*Trainer.py still identifies a cache, since an old install can be left holding nothing else, but it is a name a user's own subclass can carry and is not proof we wrote it. * Leave a shared root's own entries exactly as they are Dropping a _default or _invalid symlink is only ours to do at our own root. Where the user pointed us at a directory of their own, the entry stays and a fresh one is used instead, so no lookup follows it and nothing of theirs is unlinked. A directory that was already there also keeps its permissions: it is not marked, it is not deleted, and now it is not chmodded to 0700 either. * Give a directory one owner, and name it in the marker The marker now records the id the directory was made for. Two ids that differ only in case are one name on Windows and on a default macOS volume, so the second one gets a directory of its own instead of sharing files that either chat's delete would remove, and a delete is refused outright when the marker names someone else. A session entry that is a symlink is only unlinked at our own root; in a directory the user pointed us at it is their entry. The delete switch said anything this chat's tools wrote. A chat moved back to Recents wrote its earlier files into the project workspace, which chat deletion does not touch, so the text now says what it does. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Step around folders that were already in a shared root A tool can write anything into the directory it runs in, so a marker written there is not evidence. The answer is not to run there at all: a directory already sitting in a root the user pointed us at is stepped around, and this chat gets one of its own. Assignment goes through one claim: the directory is created and the marker taken with O_CREAT|O_EXCL under a lock, so two case-variant ids starting at the same moment cannot both take the same name. A sandbox moved up from the legacy root is claimed as part of the move. Without that an overridden root read it as someone else's and the chat could never remove it again. The terminal card now uses the same wrapper test as the adapter, and a filename holding a lone surrogate is not advertised: encodeURIComponent throws on it, so that chip could never download. * Survive a clobbered marker, and finish a move that stopped part way A marker holding anything but an id now reads as no owner rather than as somebody else's, and at our own root the chat takes its directory back. A tool writing over that file used to send its own chat to a fresh sandbox on the next launch, with everything it had made left in the old one. A destination that exists but was never claimed is a move that was interrupted, not a collision: a cross-device move copies and then unlinks, so both sides are left holding files. The rest is moved up on the next launch, and anything already above wins, the same rule whole directories follow. * Claim what we migrate, and answer only for ids a session can have A destination is claimed as soon as it is recognised as this move interrupted, so a file that could not come up no longer leaves the whole directory reading as somebody else's and the chat resolving to an empty one. In a shared root the move goes to the name the session will resolve to, unless the plain name holds nothing but a copy of what is still below, which is this same move rather than a stranger's folder. A sandbox whose claim did not take is never used: a fresh directory is made instead, since running there would put the chat inside someone else's files. The sandbox listing and download routes now refuse an id no session could have used. Everything unusable collapses into one _invalid directory, so answering for those ids handed a caller the files of every other session that had been given one. An inherited UNSLOTH_COMPILE_LOCATION= is treated as unset, rather than pinning the cache to an empty path. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Keep the record of where a chat's files are outside the sandbox Which directory belongs to which session is now written under the studio home, where tools never run, so a marker one of them deletes or writes over no longer sends that chat to a fresh directory with its files left behind. The marker stays as the on-disk claim; the record is what makes it durable, and either one is enough to delete the folder again. Migration provenance comes from that record too. It is written before the move and dropped when it finishes, so an interruption is the only thing that leaves one behind, and a directory that merely holds similar names is no longer mistaken for one of ours. An id the filesystem cannot hold gets a directory derived from it rather than a shared _invalid bucket, so those chats stop reading and deleting each other's files, and their download chips work. The routes serve them again for the same reason. The legacy move runs at startup instead of from the first read: across filesystems it copies every session, which is not something a listing on the event loop can wait for. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Separate the namespaces, and never read a directory that is not the chat's The derived _id- names are reserved: an id that already looks like one is derived too, so a literal id can no longer land on the directory of whichever unusable id hashes to it. Migration bookkeeping moved into its own object for the same reason, where no session id can name it. A read decides on what is already there, so it now checks ownership of the disambiguated name as well and returns a path under a directory we never create when the answer is no. An entry that is a symlink is stepped around whether its target is inside the root or not: claiming through one writes our marker into a directory somebody else made. The record outranks the marker when reclaiming, since all marker contents are writable from the sandbox and a valid-looking overwrite is no different from a deleted one. Chat deletion runs in a worker: right after an upgrade it also runs the legacy move, which is not something the event loop can wait on. The delete switch is offered for a chat in a project too, whose pre-move files are in its own folder. A files value that is not a list is not a wrapper, since the cards map over it. And a compiled cache in the launch directory needs a file only the compiler writes before it goes on sys.path. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Reclaim a sandbox of empty folders, and keep a linked cache usable A tool that only ran mkdir, or deleted what it wrote, left a directory tree with no files in it, which the default delete refused and no chat could reach again. Empty now means no files of the user's. A built-in cache path that is a symlink had its target removed by the clear and nothing put back, so the link dangled and the next compile could not write through it. The listing walks up to a couple of thousand entries and stats each one, so it moved into a worker like the delete paths. A persisted files array is only a wrapper if every entry has a name: the rows read it, and one null took the whole chat view down. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Trust nothing a sandbox can reach without checking it first The marker is written and read without following a link, so one replaced by a symlink no longer truncates whatever it points at, and does not read as an owner either. A cached path gets the checks a fresh resolve makes. A tool can rename its own directory and leave a link to another chat's in its place, and containment alone accepts that. A recorded directory is only used when it could have been ours: inside the root, not a link, named after this session, and not held by another one. The ledger is under the studio home, which a relative path from a sandbox still reaches, so an entry in it must not be able to name anything this chat would not have been given anyway. A recorded move target that names another root is stale: the override can change between the interrupted move and the retry. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Drop the record only once the directory is gone A delete that keeps the files was clearing the note of where they are, which after a tool has taken the marker is the only thing left saying that directory is this session's. It goes now when the removal really happened, and deletion checks ownership through the same trusted-record test the resolver uses. * Lock the ledger between processes, and stop running in borrowed folders Two Studios can share a studio home, so the read-modify-write of the ownership file takes a lock file as well as the in-process one, steals one a dead process left behind, and proceeds rather than losing a write if it cannot get it. A _default or _invalid that was already sitting in a shared root is the user's: a call with no session id used to run straight in it. We make one of our own instead, claim it and remember it, so it is the same folder on the next call rather than a new one each time. A name kept on both sides of an interrupted move is deliberate, so the move is finished with it. It was reading as failure, which left the migration flag unset and rescanned the whole legacy tree on every first tool call. An id a path segment cannot carry now rides in a query parameter: ASGI decodes %2F before it matches a route, so a slash in an API client's id arrived as a different id and a different filename. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Say in the ledger when a sandbox is in use, and stage what is arriving A call in flight is noted in the ledger as well as in memory, so a second Studio sharing this home does not rename a directory its tool is working in. A delete that arrives then is left in the ledger and carried out at the next startup, once nobody holds it. An entry older than a tool call can run is ignored, so a process that died holding one does not keep the folder for good. A cross-filesystem move is assembled under a name nothing resolves and renamed into place, per directory and per file, so a download card opened during the migration cannot read a file that is half copied. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Name it exactly, hold it per process, and finish what was staged A recorded directory has to carry a name this session could have been given, or a marker naming this session. A prefix test let a chat called a name the folder of one called ab. The busy note is one entry per process and keyed the way the in-memory count is, so two Studios in one sandbox both hold it, the first to finish does not clear the other's note, and two ids differing only in case see each other's. A delete handed over by another Studio carries the exact id and is drained when our own call ends, not only at the next startup. A move interrupted between the copy and the rename is put in place on the next launch: the legacy entry is gone by then, so nothing else would ever look at the staging directory again. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: renew sandbox leases, keep every queued delete, unblock the first tool call Busy leases are renewed while a call is still running, so a call with no timeout cannot have its folder deleted out from under it, and the read and write of the busy entry now happen under one interprocess lock. A recovered staging directory is claimed and recorded like an ordinary move, and only directories a migration of ours actually staged are promoted. Queued deletes accumulate per session instead of overwriting each other, a recorded fallback directory survives losing its marker, clearing every chat reports the ids it deleted, and the first tool call migrates only its own folder while the rest of the tree moves in the background. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: never take a link for a marker, and keep a delete from racing a tool call A cache marker reached through a symlink read as proof the directory was ours, and the cleanup deletes such a directory outright, so a link planted in an unowned unsloth_compiled_cache took the user's files with it. Ownership now needs a real file. A legacy session entry that is a symlink is left where it is: the move preserves the link, and the marker written afterwards would land inside whatever it points at, outside both roots. A first tool call migrates only its own folder and starts the rest in the background, rather than waiting out a copy of every session, and the two paths take the same per-entry lock so one session is never moved twice. Deleting a chat now cancels the generations still running for it. The in-flight guard only covers calls already inside the executor, so a request that had not got there yet could recreate the folder afterwards and write files no chat can reach. Session ids are hashed with surrogatepass, since an API client can send a lone surrogate and a POSIX name decoded with surrogateescape carries them too. File cards stream a download to the chosen path instead of buffering it into a Blob and copying that across IPC, which a multi-gigabyte artifact would not survive. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: let a delete run outside the tool lock, and offer files from every surface Deleting a chat held the lock every tool start takes for the whole rmtree, so removing one large sandbox stopped calls in every other chat. The tree is renamed out of the way while locked and removed after. A cross-filesystem move fills the destination as it goes, so a run killed part way left a partial copy that the next launch read as a session the new root already had, stranding the original. It is assembled under a name nothing resolves to and renamed into place, and a failed attempt takes its own staging tree with it. Only the sidebar dialog offers to delete the files, and after a delete from anywhere else the folder is unreachable. The route now reports which sandboxes it kept and the shared delete path offers them, rather than leaving one behind per chat. The snapshot key carries a digest for files small enough to read: on a coarse clock a rewrite of the same length inside one tick matched on mtime and size, and the call reported no file at all. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: take no directory in a shared root that we did not make The fallback name was claimed even when a directory of the user's already sat there, so a chat ran inside their files and a delete with the switch would have removed them. Both names now get the same test, and a fresh one is taken rather than anything already present. A marker reached through a symlink counted as ownership, since isfile follows one, which made an unrelated directory deletable. The path a chat with nothing of its own resolves to now lives outside every sandbox root, so a folder the user keeps at that name is never listed or served. A request-path migration goes through the resolver like the whole-tree pass, rather than stopping at the plain-name collision and leaving the chat with an empty sandbox and its files at the old root. Deleting a project now takes the sandboxes of the chats it held, which are otherwise left with no record pointing at them. Native file cards ask for an absolute loopback URL, which the native download command requires: a relative one was rejected before the request was made. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: find fallback sandboxes by marker, repair clobbered markers, sweep interrupted deletes - a fallback with a name nothing can recompute is found again by its marker, on a later launch, on a read and on a delete - a marker tool code removed from a directory this run claimed is written again rather than the directory being read as somebody else's - an interrupted detached delete is finished on the next launch, ours only - a pre-upgrade chat whose id starts with the derived prefix is migrated from its literal legacy directory name - clear-all reports the sandboxes it kept and offers to delete them, through the same helper the sidebar delete uses * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: fix the Python tool image path test on the new sandbox route The module import needed the .ts extension the node runner resolves, and an id with a path separator now travels in the query rather than in the path. * Studio: keep the file envelope ours, and land every sandbox move somewhere free - the fallback suffix is encoded like the name it is appended to, so an id with a lone surrogate can still step aside - a legacy move picks a free target when both derived names are the user's - a read finds a marked random fallback, not only creation and deletion - a chat started during clear-all is cancelled before its sandbox goes - a call no longer reports another in-flight call's scratch script - a marker line a program printed itself is broken before ours is appended * Studio: report what a delete kept, and never delete the only copy of a move - a removal deferred behind a running tool call is reported as kept - a project delete uses the membership its transaction deleted, cancels the late ones, and returns the member sandboxes it preserved - a staged legacy move that fails its final rename keeps the tree it already moved, marked, and puts it back at the legacy root when it can - closing an incognito chat cleans up the sandbox its tool calls created - a symlinked directory counts as a file of the user's * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: bound the snapshot hashing, and claim only what one call wrote - a snapshot stops hashing after 64 MiB and falls back to mtime and size - a file written while two calls shared a workdir is claimed by neither - a read serves the legacy directory while the background move is unfinished - a markerless sandbox is only deletable by the id its name carries * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: a shared workdir claims nothing, and one sweeper does the deleting - a call that ran alongside another in the same workdir reports no files, with no dependence on filesystem timestamps - a delete migrates its own session rather than the whole legacy tree - the file envelope is read only from the tools that emit one - a deferred removal runs after the global session lock is released - a staged migration that could not be moved in is adopted by the resolver - detached trees go to one sweeper thread instead of one per chat * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: one migration lock per session, and a sentinel path that cannot escape - a chat's legacy copy no longer blocks another chat's first tool call - the empty-sandbox sentinel derives its name, so an absolute session id cannot make it resolve to a directory of the system's - a legacy copy whose destination is already this chat's is left where it is - an interrupted delete at our own root is swept without needing its marker * Studio: derive every fallback name, and delete the project the transaction did - fallback names are derived and recomputable, so a big shared root can no longer hide a chat's folder from the scan - the startup migration picks a free target like the request path does - a staged move is marked before the rename, not after - a delete uses the directory this process created when its marker is gone - the project delete acts only on the membership its transaction deleted - the client reads the file envelope only from the tools that emit one * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: adopt a sandbox by name, and keep what the old shared bucket holds - a directory is only adopted under a name derived from the chat's own id, so a marker a tool rewrote cannot hand one chat another chat's files - the pre-upgrade _invalid bucket is read where it is and never moved - a user's own .unsloth_sandbox file is preserved by the migration - a project workspace is removed after its chats' tool calls are stopped - a sandbox a surviving fork still shows cards for is kept - the second research path names its tool when stripping * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: wait for a project's tools, and match a sandbox reference exactly - the workspace delete waits for the member chats' tool calls to unwind - a reference is a decoded sessionId value, not a substring of the message - a workspace a surviving fork still shows cards for is kept - the project delete cancels the research runs its transaction removed, by id * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: hold a session closed while its sandbox goes, and stop the right tools - a start for the session being removed waits, so no call is handed a folder the removal is about to rename away - the project wait covers project-<id>, which is what its tool calls run as - a kept project workspace still resolves after the row is gone - a marker-named file is preserved unless it is the exact marker being written - clear-all cancels the research runs its transaction removed, by id - the supervisor is signalled even when the run row has already cascaded * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: remember a kept workspace, and never delete one still in use - a workspace whose idle wait timed out is kept, not removed under a live call - a kept workspace is written down, so a custom location still resolves and the next delete collects it once nothing points at it - a marker a tool rewrote with another id is corrected, its file preserved - the empty-sandbox scan runs with the global lock released - a compiled-cache directory needs a generated file, not just the name * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: record a kept workspace either way, and finish the delete it promised - the path is written down whether or not files were to be deleted, since the row that knew it is gone in both cases - the record says whether the user asked for the files, and only those are ever collected - a delete held up by a running tool call finishes when that call ends, and the startup sweep picks up one a kill interrupted * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: hide only the sandbox's own bookkeeping, and keep the records writable - a pending workspace delete is collected after any deletion, not only one that asked for files - a nested file named like the marker is an ordinary file in both walks - the orphan records live under the studio home, which is ours to write - an inline image below a subdirectory keeps its path separators * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: keep the old shared bucket to the ids that shared it - only an id the previous code could not use as a name reads _invalid, so an ordinary chat is never served another chat's files - a markerless directory is this chat's only when its name says so, on the read path as well as the delete path - a workspace delete that failed stays pending instead of being forgotten - a deferred workspace delete removes the whole project workspace, as the immediate one does * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: keep a retry record, fail safe on a locked database, claim the default sandbox * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: tighten the sandbox comments * Studio: serialise a legacy read with its move, retry a detached delete, keep an overwritten marker * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: reserve the fallback names, recheck a shared call, and keep a recreated chat's files * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: validate every recorded delete, pin the download to the file it checked, and reclaim a fork's source * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: a linked root is the user's, and a recreated chat or project keeps its files * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: answer the download probe, and key kept-folder records by kind and id * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: keep a directory this run claimed, list the root once, and unwrap only our own tools * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: keep the sandbox wrapper check a type guard * Studio: recover on startup, keep a reused project id from stranding the old workspace * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: final pass over the sandbox comments * Studio: skip the snapshot digest where mtime already separates the writes --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-08-10 01:11:46 -07:00
Michael Han	a00fe86c13	Studio: read model text as utf-8 so umlauts survive on Windows (#7467) ... * Studio: read model text as utf-8 so umlauts survive on Windows Chat rejects or mangles non-ASCII on Windows: "ä ö ü" in a prompt, a chat template, or a model path comes back as mojibake, or the load dies with UnicodeDecodeError. open() and Path.read_text() fall back to locale.getencoding() when no encoding is passed. On Windows that is the ANSI codepage (cp1252, cp932, cp1251, ... by system locale), never UTF-8. Hugging Face writes these files as raw UTF-8, so every read of one decodes with the wrong codec: - tokenizer_config.json, which holds the chat template. Templates routinely carry -> arrows, smart quotes and CJK, so this is the common path into chat - config.json and adapter_config.json - modules.json, Ollama manifests, and the .py sources the remote-code scanner reads before a model is allowed to load The llama-server and embedding-server stdout readers have the same problem via subprocess(text = True); they now decode utf-8 with errors = "replace" so a stray byte cannot kill a log reader. Encoding arguments only, no logic changes. tests/test_chat_text_encoding.py covers a config.json and a chat template holding umlauts, arrows and CJK, plus the remote-code scanner reading a source file with umlauts. Those pass anywhere the locale is already UTF-8, so a fourth test re-runs the readers under -X warn_default_encoding and fails on any platform if an encoding argument goes missing again. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: name utf-8 explicitly on the remaining text I/O, with an AST guard (#7465) * Studio: name utf-8 explicitly on the remaining text I/O Follow-up to the model-text reads in #7467, covering the rest of the backend: system probes (nvidia-smi, amd-smi, powershell, git, node), package installers, /proc and /sys readers, and internal marker files (pid, install id, bootstrap password, Colab credentials). Same reason as #7467. open(), Path.read_text()/write_text() and subprocess(text = True) fall back to locale.getencoding(), which on Windows is the ANSI codepage rather than UTF-8. These paths are mostly ASCII today, so this is hardening, not a live bug. Encoding arguments only, no logic changes. Adds tests/test_text_io_encoding.py: an AST guard walking every backend source and asserting text I/O names its encoding, so the class of bug cannot creep back in one call at a time. 275 files. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Catch aliased subprocess and positional Path.open, migrate legacy JSONL The guard only matched a receiver literally named subprocess, so worker.py's `import subprocess as _sp` hid three text = True installs that decode pip output with the ANSI codepage. It also skipped any .open() with more than one positional argument, though Path.open takes buffering/encoding/errors/newline positionally. Resuming a scrape written by an older release is the other half: those JSONL lines are in the locale codepage, so the UTF-8 preload raised, the dedup keys were silently forgotten and duplicates were appended to a now mixed-encoding file. Decode with the locale codepage as fallback and rewrite as UTF-8 before the append handle opens, since Windows cannot replace a file it holds open. * Stream the JSONL preload and keep a torn line from relabelling the shard Reading the whole shard to migrate it was wrong twice over. These files reach gigabytes on a large scrape, so the preload now streams line by line and the rewrite streams through a temp file. Worse, one interrupted append used to condemn the file: the whole-file UTF-8 decode failed, every byte was retried as cp1252, and the rewrite persisted mojibake over records that were fine. A line now counts as legacy only if the locale codepage both decodes it and yields valid JSON, which a torn UTF-8 line does not. Damaged lines are skipped and copied through byte for byte. When the rewrite cannot be written at all, the append handle opens with the legacy encoding rather than mixing UTF-8 into the file. install_wheel takes run = subprocess.run as a parameter, so the guard cannot see it. Both wheel installs there now name their encoding. * Decide the shard's encoding from the file, not one line at a time Some byte strings parse both ways. cp1251 `Р°` is D0 B0, which is also valid UTF-8 for `а`, so a UTF-8-first parse quietly showed the wrong text instead of migrating it. A line now yields both readings, and the file decides. Any line that parses under the codepage but not as UTF-8 is unambiguous evidence, and ambiguous lines then follow that verdict, which is enough for any real shard: ordinary Cyrillic or Japanese prose is invalid UTF-8 several times per line. Keys for ambiguous lines are re-derived from the legacy reading during the rewrite. A shard is undecidable only if every line is ambiguous, and nothing can tell those apart. latin-1 is also tried after the locale codepage, so a scrape carried from Windows to a UTF-8 machine still has a reading rather than none. Requiring valid JSON, not just a decode, keeps that from claiming torn lines. * Weigh the whole shard, and never lose a record on the fallback path One structurally valid JSON line carrying a stray 0x96 parses as cp1252, so a single-line verdict let it relabel a healthy shard and mojibake every good record in it. Each line with non-ASCII bytes now votes: parsing only under the codepage is evidence for legacy, parsing as UTF-8 is evidence against, since codepage text rarely forms valid multibyte UTF-8. Ties leave the file alone. When the migration cannot be written the append handle uses the legacy codepage, and errors = "replace" quietly turned characters it cannot hold into question marks while write() still reported success. That path now escapes to \uXXXX instead, which is ASCII, so every codepage holds it and json.loads returns the exact characters. Nothing needs replacing, so errors = "strict" is safe. stream_installer runs sys.executable, so its output is now decoded as UTF-8 by utf8_child_env rather than read as the ANSI codepage. * Only rewrite a shard we can attribute, and append ASCII when we cannot latin-1 was doing too much work. It reads any byte, so it gave a moved shard a reading, but it is the right text only for cp1252: cp1251 Привет came back as Ïðèâåò and the rewrite made that permanent. The codepage is now trusted only when it is the locale's, and an untrusted reading is never written back. That leaves three cases where the file holds bytes UTF-8 cannot read and we are not converting it: no codepage to attribute it to, ambiguous lines outvoting the unambiguous ones, and a preload that could not read the file at all. All three used to append UTF-8 into it. They now append pure ASCII, which every ASCII-compatible codepage stores identically, so the file keeps decoding exactly as it did and no record is lost. Keys from the two readings are also kept apart. A damaged line in a healthy shard was marked seen through its codepage reading, so the retry that would have replaced the unreadable record was refused as a duplicate. * Let the flash-attn install stub take the kwargs the installer now passes _run_kwargs gained encoding and errors, so the one stub in this file that spelled its signature out rejected the call. The other four here already take **kwargs; this one now matches. * Do not let a stuck temp file mask the migration failure unlink() on the failure path could raise in its own right, on a stale .utf8.tmp directory or a temp another process holds. That escaped the constructor instead of returning False, so the caller never reached the ASCII append fallback that keeps the shard single-encoding. The pip fallback in install_wheel also spawns a Python child, so it gets utf8_child_env like the probe above it already had. The uv and nvidia-smi children are native binaries, where PYTHONIOENCODING would do nothing. * Stop converting legacy shards; the encoding that wrote them is unknowable trusted only ever meant that the bytes parse under this machine's codepage, which for a single-byte codepage is nearly always true. A cp1251 shard opened on a cp1252 Windows box decodes cleanly and would have been rewritten with Привет as Ïðèâåò. That is the fourth way this rewrite could corrupt a shard, and the common cause is that a file's encoding cannot be recovered from its bytes. So the rewrite is gone. The shard is left exactly as found, and appends are pure ASCII whenever it holds bytes UTF-8 cannot read, which is what actually delivered the no-mixed-encoding guarantee the rewrite was added for. Dedup keys still come from whichever reading parses, since ids are ASCII either way. This also removes the temp file, so there is no longer any file mode or ACL to carry across. * Scan the sandbox shim; it is shipped code, not a build artifact sandbox_site is on the sandboxed child's PYTHONPATH for every Python run (tools.py:332, 2660), so excluding it let two unannotated text calls through in code we ship. Both read and write the remap sidecar, which holds file paths. The exclusion list is meant for build output only, so the directory comes off it and the two calls name their encoding. * Force the worker's pip children to UTF-8, and read DBCS keys with a DBCS codec The three installer calls run sys.executable -m pip with an inherited environment, so the parent decoded UTF-8 while the child emitted the ANSI codepage. They now go through utf8_child_env like the other Python children. Two tests asserted no env kwarg was passed as a stand-in for no HIP flag being injected. They now assert the flag itself, which is the guarantee they were written for and does not depend on how the env is delivered. Separately, latin-1 cannot stand in for a double-byte codepage while recovering dedup keys: cp932 表 is 95 5C, and the trail byte reads as a JSON backslash, so the record failed to parse and its id was forgotten, appending a duplicate on resume. cp932, cp936, cp949 and cp950 are tried too. The reading is still only ever used for keys, which are ASCII and identical whichever codec parses. * Require more than one legacy line before trusting its dedup keys A shard whose valid records are all ASCII casts no UTF-8 votes, so a single damaged line won the vote by itself, its key was remembered, and the retry that would have replaced the unreadable record was refused. One such line is genuinely undecidable: a legacy record with one accented character and an ASCII record with one stray byte are the same shape. Reading it as damage costs a duplicate; reading it as legacy loses the record for good. Only one of those is recoverable, so it is now read as damage. A real legacy shard has a legacy line for every record carrying an umlaut, so its dedup is unaffected. * Append ASCII whenever the shard already holds non-ASCII bytes The gate asked whether any line was undecodable as UTF-8, which misses a shard where every legacy line happens to be valid UTF-8 too. A cp1251 shard of Р° records is bytes D0 B0 throughout, so appending 世界 as UTF-8 left a file where cp1251 reads the old records correctly and the new one as mojibake, and UTF-8 does the reverse. No single decoding recovered the whole scrape. The gate is now simply whether the shard holds any non-ASCII byte at all, which covers both cases and is easier to reason about: if what is already there reads differently under different encodings, do not add more bytes that do. Appending ASCII costs only \uXXXX escapes, which json.loads turns back into the exact characters, and it leaves the new record correct under either reading. * Skip the two Linux-gated flash-attn tests off Linux _should_try_runtime_flash_attn_install ends in sys.platform.startswith( "linux"), and the threshold test one line above already asserts exactly that, so the two tests that drive _ensure_flash_attn_for_long_context past the gate cannot pass anywhere else: the call returns before it reports a status. They were written on Linux and only surface once the suite actually runs on Windows or macOS, where both fail on an empty status list. This PR is about making the backend behave on Windows, so its own suite should be runnable there. * Fail closed when a KFD topology node does not decode This PR pins that read to utf-8, which turns an undecodable byte into UnicodeDecodeError. That is a ValueError, not an OSError, so it slips past the handler one line below and escapes a helper whose docstring promises to fail closed on any unreadable node. The caller would then lose the whole HIP-order map on a machine that has AMD GPUs, and the reason the helper fails closed is that dropping a node shifts every later ordinal and lets a similar-capacity GPU pass the total-size guard while showing another card's usage. Widening the handler is the same one-line change main already made in #7487, so the two agree and the eventual merge is clean. * Tighten the comments added in this branch * Treat an undecodable marker and undecodable metadata as malformed, not fatal Two more places where pinning the decode changed the failure mode. A UnicodeDecodeError is a ValueError, so neither `except OSError` nor `except (JSONDecodeError, OSError)` catches it, and both sites had a documented fallback that stopped being reached. An undecodable .transport marker used to read as an unknown value, and the caller then safely purged and restarted the partial download. It now aborts prepare_cache_for_transport instead, so the transfer fails rather than retrying. Undecodable .meta.json used to fall back to the file's own name, the same way invalid JSON does. It now aborts URI construction for the entire unstructured seed, so one corrupt byte in original_filename takes out the whole dataset. Both handlers are widened, matching the KFD fix earlier on this branch. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Widen two more decode guards, and pin the kernel installer's pipe Same shape as the ones already fixed here: the read was pinned to UTF-8 while the handler around it still only catches OSError, and UnicodeDecodeError is a ValueError. hf_cache_snapshot_dir answers whether a model is already on disk, and the offline embedding checks turn a raise into a 500. A torn refs/main used to decode into a nonsense commit and miss the snapshot dir; it now skips that cache root and keeps looking. _remove_pid_file runs first in _graceful_shutdown, so a corrupt studio.pid raising there abandoned the inference, export, training and tunnel children the rest of that function exists to kill. ssm_runtime's source-build path builds its subprocess kwargs in a dict and splats them through _run_with_heartbeat, so neither the encoding guard nor the earlier sweep saw the text = True in it: pip's output was still decoded with the Windows ANSI codepage, where a non-ASCII path or a compiler diagnostic mojibakes or raises over an install that was going fine. It now pins the same utf-8/replace pair install_wheel uses, and the HIP branch extends that env rather than replacing it. The guard learned the dict-literal shape and reddens on the old code (ssm_runtime.py:253). * Tighten the comments around the UTF-8 text I/O pins Collapse the multi-line rationales added with the encoding pins down to a line or two each, drop what the code already says, and use one wording for the repeated child-env note. * Do not let an unreadable bootstrap password stop startup, and narrow the kwargs guard ensure_default_admin calls _load_bootstrap_password for every existing admin and the lifespan calls that with no handler, so pinning the decode turned a damaged or pre-pin .bootstrap_password file into a backend that will not start. We write that file ourselves in UTF-8, so a byte that will not decode belongs to a file whose plaintext is worthless anyway; it now reads as no bootstrap password, the same answer as an absent file. A readable one still loads. The new kwargs check also judged every dict literal in the tree, so an unrelated payload carrying "text": True would have been reported as subprocess configuration with a misleading message, and a dict that fills in its encoding on a later line would have been reported too. It now only judges a dict that actually reaches a call, either splatted through a name or written at the call site, and treats a later kw["encoding"] assignment as satisfying it. The ssm_runtime shape it was written for is still caught, and a test pins both directions. * Stop reading a UTF-8 record a second time _read_line always parsed the line under the codepage as well, even when it had already read as UTF-8. Both callers take the UTF-8 reading when there is one and never look at the other, so on a healthy shard the second parse is pure waste, and this file reads all of one on every resume of a scrape it expects to reach gigabytes. Measured on 200,000 records, 76 MB: 1.96s before, 0.81s after, so the double reading was costing 2.8x. The early return is limited to a record, since the key lookup deliberately falls through to the codepage reading when UTF-8 yields something that is not one. A line UTF-8 cannot read still tries the codepage, latin-1 and the double-byte encodings as before, which is what the second reading is for. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Pin the scanned source fixture's line endings test_remote_code_scan_reads_non_ascii_sources compared a file's contents against the string it wrote, but wrote it in text mode, so Windows translated the line ends on the way out and the read back differed by a carriage return. That is the writer's doing, not the encoding the test is about, and it was the one failure on the Windows runner that belonged to this branch. The fixture now writes with newline = "" so the bytes on disk are the string on every platform. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Trim the newer comments to their point Shorten the widened-guard and state store notes added since the last pass, and collapse the line-ending note on the scanned source fixture. * Read the scraper checkpoint as UTF-8 only, never as a codepage A checkpoint holds nothing but base64 cursors and booleans, so one written by an older locale-encoded release is byte-identical to a UTF-8 one and already reads back. The codepage fallback can therefore only ever contribute non-ASCII: if a single-byte reading of the file were all ASCII, the UTF-8 read would have succeeded first. So the only file it changes the answer for is a damaged one, and there it turns a safe reset into a resume on a mojibaked cursor. GitHub answers that with INVALID_CURSOR_ARGUMENTS at HTTP 200, gh_client returns the partial document, and the scraper reads zero nodes and an empty pageInfo, which marks the stream done. Every later resume then skips it entirely. Reading UTF-8 only restores the earlier behaviour of dropping a checkpoint that will not decode, which re-scrapes from the first page while the writers dedup the replay. The shard scan below keeps its codepage reading; those records do carry non-ASCII. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Gate the remaining tilelang install tests to Linux _tilelang_platform_supported() returns False off Linux, so _ensure_tilelang_backend returns before the install and the subprocess mock these six assert on is never called. They fail on macOS runners for that reason alone. The rest of the file already carries this marker; these were missed. * Gate the Windows-incompatible worker and ROCm tests Two different gates, because the production code has two. The causal-conv1d and flash-linear-attention installers bail out on sys.platform == 'win32' alone and run everywhere else including macOS, so those cases get not_on_windows; marking them linux_only would skip tests that legitimately pass off Linux. The DRM and KFD readers return early unless platform.system() is Linux, and their fixtures build a fake sysfs tree needing PCI addresses like 0000:00:02.0 as directory names, which Windows cannot represent, so those get linux_only. The two visible-utilization cases failed for a different reason: on Windows get_visible_gpu_utilization takes the AMD adapter branch ahead of the torch fallback under test, and probing it imports torch, which the runner lacks. Stubbing that branch empty leaves every other platform unchanged. * Treat unparseable JSON nesting as a parse failure, and guard os.fdopen json.loads answers nesting it cannot descend with RecursionError, a RuntimeError, so _parse let it escape where the catch-all it replaced discarded the record. Both callers run _parse outside any further handler, so one damaged checkpoint or shard line aborted the scraper at startup. The encoding guard also missed os.fdopen, which is open() on a descriptor and takes the same locale default in text mode. It flags exactly the two text-mode calls that were left unencoded; the swap lock file's reader was already pinned to UTF-8 while its writer still used the codepage. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Write the non-ASCII source fixture without a 3.10-only argument Path.write_text() only grew newline in 3.10, and pyproject declares requires-python >=3.9, so this raised TypeError there. open() takes the same argument on every supported version and pins the bytes on disk the same way. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Tighten encoding comments * Follow subprocess calls through callable aliases in the encoding guard --------- Co-authored-by: Unsloth <michaelhan@Michaels-MacBook-Pro.local> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: danielhanchen <unslothshared@gmail.com> --------- Co-authored-by: Unsloth <michaelhan@Michaels-MacBook-Pro.local> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: danielhanchen <unslothshared@gmail.com>	2026-07-28 21:27:27 -07:00
Daniel Han	1daaa5cbb4	Let a decode failure degrade instead of escaping a fail-closed helper (#7487) ... * Let a decode failure degrade instead of escaping a fail-closed helper Pinning utf-8 makes a read that used to return mojibake on Windows raise instead. 33 of those reads sit under a handler catching OSError or json.JSONDecodeError but not UnicodeDecodeError, which subclasses ValueError, so a corrupt file would now escape a helper written to return a default. Adds UnicodeDecodeError to those tuples only. * Treat an undecodable install lock as stale instead of retrying forever	2026-07-27 03:26:08 -07:00
Daniel Han	3fd948eb95	Pin utf-8 on shipping-code text I/O instead of the operator locale (#7486) ... * Pin utf-8 on shipping-code text I/O instead of the operator locale 113 read_text/write_text/open call sites across unsloth, studio and unsloth_cli let locale.getencoding() decide the encoding. That is utf-8 on the Linux and macOS runners and cp1252 on a stock Windows install, so the same file decodes differently for a Windows user and silently produces mojibake or raises UnicodeDecodeError. Adds tests/test_runtime_text_encoding.py to keep it that way. It resolves openers through each file's own imports rather than a fixed list of module names, so an aliased tarfile.open or a local from PIL.Image import open is not asked for an encoding it does not take. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Scan tracked files only and resolve the unbound Path calling forms * Honour PEP 263 when scanning sources and migrate a legacy JSONL before appending * Scope guard imports lexically and only migrate a legacy file when it round-trips * Leave a legacy JSONL untouched and resolve path aliases in the foreign-opener check * Tighten comments --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-07-27 02:14:20 -07:00
oobabooga	dbb06ff60e	Studio: add configurable model download location (#7274) ... Adds a configurable Hugging Face model download cache location to Unsloth Studio, selectable from Settings, with per-cache download manifests, scoped deletion, and read-only inventory of previously selected caches.	2026-07-23 01:34:38 -07:00
Nilay	95d9970233	persist llama.cpp KV cache across idle auto-unload (slot save/restore) (#7204) ... * Studio: persist llama.cpp KV cache across idle auto-unload (slot save/restore) * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: address KV persistence review feedback * Studio: guard KV restore on launch config * Studio: fix KV resume purge race, fingerprint requested ctx, purge on disable * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: re-check idle/keep-KV settings after slot save, ns file identity * Studio: shard-aware KV guard, honor user --no-cache-prompt, early save cap * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: honor LLAMA_ARG_CACHE_PROMPT env in slot-save guard * Studio: derive prompt-cache state from final argv for slot saves * Studio: stat LoRA/control-vector sidecars in KV restore fingerprint * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: parse csv and FNAME:SCALE sidecar syntax in KV fingerprint * Studio: address codex review on idle-unload KV resume * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: harden slot-save cleanup, cap accounting, stale-KV guard, save timeout * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: treat unavailable KV estimate as full-cap for slot-save disk check --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Daniel Han <danielhanchen@gmail.com>	2026-07-20 00:12:42 -07:00
Michael Han	6d8c18cd1a	Replace standalone Studio wording with Unsloth (#7221) ... * Replace standalone Studio wording with Unsloth Replace the single word Studio with Unsloth wherever it is used as shorthand for Unsloth Studio in docs, CLI output, UI strings, i18n locales, workflow display names, comments and docstrings. Kept unchanged: the full name Unsloth Studio, third party product names (LM Studio, Visual Studio, Mac Studio), feature names (Recipe Studio, Fine-tuning Studio and its translations), and all identifiers such as env vars, commands, paths and filenames. * Address review feedback on the Studio wording rename Use "an" before Unsloth where the rename left the article as "a". Restore the split brand where Unsloth and Studio render as two halves of the full product name: the onboarding sidebar subtitle and the IPv6 localhost warning. Scope two messages to the full name Unsloth Studio where plain Unsloth was misleading: the AMD README bullet and the CLI studio setup error.	2026-07-19 00:47:04 -07:00
Gaurav Dubey	dc65638b7d	Studio: expose Windows drive roots in the folder browser (#7082) ... * Studio: expose Windows drive roots in the folder browser The model-selection folder browser bounds navigation to the roots returned by _build_browse_allowlist(), which exposed Linux removable-media mounts via linux_run_media_mount_roots() but had no Windows analog. As a result a user on C: could not browse to D:/E: to pick a model directory. Add windows_drive_roots(), a Windows-only companion to linux_run_media_mount_roots() that lists readable logical drive roots, and wire it into both browse-allowlist builders and their suggestion chips so other drives are both navigable and offered as quick-picks. The helper is a no-op on Linux/macOS, so existing platforms are unaffected. Closes #6368 * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: cover the Windows drive-root browse wiring with an integration test Add an allowlist integration test mirroring the Linux side's test_legacy_browse_allowlist_includes_linux_run_media_mounts: it extracts _build_browse_allowlist from routes/models.py, stubs external_media so windows_drive_roots() yields a fake drive root, and asserts that root becomes browsable through the built allowlist. Proves the wiring, not just the helper. * Studio: skip inactive drives via GetLogicalDrives before probing Resolve active logical drives from GetLogicalDrives() before probing each letter with os.path.isdir. Probing a drive letter mapped to a disconnected network share can otherwise block the async backend for tens of seconds per letter. The call degrades gracefully (falls back to probing all letters) when ctypes/windll is unavailable, so behavior is unchanged on Linux/macOS. Tests override the bitmask source to stay deterministic on real Windows hosts. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: allow browsing descendants of a drive-root allowlist entry routes/models.py _is_path_inside_allowlist() checked descendants with startswith(root_real + os.sep). A drive root ("D:\") already ends in a separator, so the prefix became "D:\\" and a child like "D:\models" was rejected with 403 after the browser opened the drive root. Only append a separator when the root does not already end in one. folder_browser.py already uses commonpath and was unaffected. Adds a regression test covering the separator-terminated-root descendant case. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: enforce the system-directory denylist during folder browsing Exposing whole Windows drive roots (and any legacy-registered filesystem root) widened the browse allowlist above system directories, but the browse resolvers only re-applied the credential/config denylist, not the _denied_path_prefixes() system-dir denylist that scan-folder registration enforces. That let browse-folders enumerate C:\Windows, C:\Program Files, /etc and /proc. - Add is_denied_system_path() to both storage modules and enforce it in both browse resolvers (legacy routes/models.py and hub folder_browser.py), on each resolved child and on the final target, keeping the /run/media carve-out. - Rework the legacy _is_path_inside_allowlist to use splitdrive + commonpath so a Windows drive root authorizes its descendants while a bare POSIX / does not, and to compare case-insensitively like the hub browser. - Reject the filesystem root in the legacy add_scan_folder, matching the hub. - Hide denied system dirs from browse listings and suggestion chips. - Add tests/test_browse_denylist.py and update the external-media path tests. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: make browse-denylist tests OS-portable The browse-time denylist tests used real /etc and tmp_path locations; on macOS tmp lives under the (legitimately denied) /private/var and /etc resolves to /private/etc, so three tests failed there. Pin the platform / use a tmp-based denied prefix so they assert the same behavior on Linux, macOS and Windows. * Studio: apply the bare POSIX-root guard to the hub folder browser too The _is_path_inside_allowlist guard that stops a legacy-registered '/' scan folder from authorizing every absolute path lived only in the legacy browser. The hub browser used commonpath without it, so a stale '/' row let it descend into /var, /root, /home -- which the system-directory denylist (/proc /sys /dev /etc /boot /run) does not cover, while the legacy browser blocked them. Mirror the legacy guard so both browsers treat '/' identically. Also resolve each directory entry before the denylist check in both listing loops, so a symlink or junction pointing into a denied dir is hidden instead of rendered as a row that 403s on descent. Adds legacy-vs-hub parity tests. * Studio: bound Windows drive probing so a disconnected mapping can't stall the browser GetLogicalDrives includes mapped network drives, so a disconnected but still mapped drive (e.g. Z: -> \\nas\share) stays set in the bitmask and reaches os.path.isdir, which can block for tens of seconds while Windows tries to reconnect. Because windows_drive_roots() runs synchronously while building both folder-browser responses, one stale mapping stalled every browse request. Probe each surviving drive in a daemon thread bounded by a short timeout and skip it if it does not answer in time, so a hung mapping is dropped instead of blocking the caller. Connected drives (local or network) still respond well within the timeout, so drive discovery is unchanged. Corrects the GetLogicalDrives docstring, which claimed the bitmask alone prevented the stall. * Studio: probe drive/media roots once per browse request, not twice Both folder browsers called windows_drive_roots() (and linux_run_media_mount_roots()) twice per browse request: once to seed the allowlist in _build_browse_allowlist() and again to build the suggestion chips. With the bounded drive probe, a disconnected mapped network drive then paid the timeout twice per folder click. Probe both once in the request handler and pass the results into _build_browse_allowlist(), reusing them for the chips, in both the legacy and hub browsers. Adds a test asserting the roots are reused, not re-probed. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: run the legacy browse endpoint in the threadpool, fix its stale test Two follow-ups from review of the drive-probe changes: - browse_folders was 'async def' but does only blocking filesystem I/O (the timeout-bounded drive probe, iterdir, realpath). On the event loop a disconnected mapped drive waiting out its probe timeout stalled every other request. Declare it sync 'def' so FastAPI runs it in the threadpool, matching the hub browse endpoint. No await was used in the body. - test_browse_folders_hides_sensitive_dirs monkeypatched _build_browse_allowlist with a zero-arg lambda; the once-per-request refactor now calls it with (media_roots, drive_roots), so the lambda raised TypeError. Accept and ignore the args. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: probe Windows drive roots concurrently so multiple dead mappings don't stack timeouts windows_drive_roots() probed each candidate serially, so N disconnected-but-mapped network drives each paid the full per-drive timeout in turn (e.g. four stale mappings added ~8s to every folder-browser request). Collect the candidate roots first, then probe them all at once under a single overall deadline, so the added delay stays at ~one timeout regardless of how many drives are disconnected. _readable_dir_within stays as a thin single-path wrapper for its existing callers/tests. * Studio: tighten comments in the folder-browser drive-root changes Condense the comments and docstrings added by the Windows drive-root and system-directory denylist work to be shorter and clearer while keeping the security and correctness rationale intact. Comment and docstring text only; no code changes. * Studio: iterate the input, not the results dict, when collecting readable drive probes _readable_dirs_within returned {path for path, ok in results.items()...}, but a probe thread that exceeded the join deadline is still alive and can insert its key into results during that iteration, raising 'dictionary changed size during iteration' -- reachable exactly in the disconnected-mapped-drive case the probe exists for. Iterate the fixed input list and read results.get(path) (an atomic read) instead. * Studio: keep the browse-route containment tests denylist-inert so they pass on macOS test_browse_folders_route.py exercises allowlist containment and the file-vs-directory guard, not the system-directory denylist. On macOS pytest tmp_path resolves under /private/var, a denied prefix, so _resolve_browse_target 403s the fixture dirs before the containment logic runs (4 failures). Add an autouse fixture that makes is_denied_system_path inert in this file; the denylist keeps its own coverage in test_browse_denylist.py. * Studio: keep the hub browse tests denylist-inert so they pass on macOS * Studio: register a UNC share root; only reject local filesystem roots * Studio: reject device drive roots and browse a registered UNC share root * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: treat device-namespace volume GUID roots as local filesystem roots --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: danielhanchen <danielhanchen@gmail.com>	2026-07-15 00:24:11 -07:00
ramisworld	01f7e14988	Fix Studio custom folders on Linux external drives (#6799) ... * Fix external drive custom folder selection * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update studio/backend/tests/test_linux_external_media_paths.py Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Keep legacy media scan validation strict * Apply sensitive-dir denylist to legacy folder browser for PR #6799 The legacy /api/models browse endpoint gained the new /run/media mount roots in its allowlist but not the credential/config guard that scan-folder registration and the Hub browser already enforce. Filter sensitive names during enumeration and reject them in _resolve_browse_target so .ssh, .aws, .config, etc. under allowlisted roots stay unbrowseable, matching the Hub browser. Add a public contains_sensitive_path_component helper and cover the legacy resolver with a regression test. * Trim redundant comments in PR #6799 changes * Skip sensitive Linux media roots * Reject sensitive dirs at exact browse roots for PR #6799 Both _resolve_browse_target functions only checked contains_sensitive_path_component while walking descendant parts, so requesting an allowlisted root itself (empty relative path) returned it unchecked. A pre-existing scan-folder row under ~/.ssh, ~/.aws, ~/.config, etc. (registerable before the denylist was added) is re-added to the allowlist on upgrade and could then be browsed. Check the resolved target once before returning in both the legacy and Hub browsers, and cover the root case in both test suites. * fix: avoid unused path helper reexports * fix: import sensitive path helpers directly * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> Co-authored-by: Daniel Han <danielhanchen@gmail.com> Co-authored-by: imagineer99 <samleejackson0@gmail.com> Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>	2026-07-03 19:10:04 +01:00
Michael Han	7bd8e64921	Studio: honor custom HF_HOME for model download and load (#6510) ... * Studio: honor custom HF_HOME for model download and load _setup_cache_env always derived HF_HUB_CACHE and HF_XET_CACHE from XDG_CACHE_HOME / ~/.cache, ignoring a user-set HF_HOME. Because it sets HF_HUB_CACHE explicitly and that variable takes precedence over HF_HOME in huggingface_hub, the hub cache was pinned to the standard location: a model already present under a custom HF_HOME was detected but then re-downloaded from scratch on load. Seed HF_HUB_CACHE and HF_XET_CACHE from HF_HOME when the user set it (HF's own default is $HF_HOME/hub and $HF_HOME/xet), and honor the legacy HUGGINGFACE_HUB_CACHE alias. The hub download workers call snapshot_download without a cache_dir for both the Xet and HTTP-fallback paths, so they follow HF_HUB_CACHE; fixing it here unifies detection and both transports on one root. Explicit HF_HUB_CACHE / HF_XET_CACHE stay untouched. Adds tests for the custom-HF_HOME, default, explicit-override, and legacy-alias cases. Fixes #5182. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: do not crash startup when a custom HF_HOME is not writable Seeding HF_HUB_CACHE/HF_XET_CACHE from HF_HOME means _setup_cache_env now mkdir's under a user-controlled path. A non-writable or not-yet-mounted HF_HOME (typo, offline drive) would raise and crash startup, where the old code silently fell back. Make the mkdir best-effort; the env var is still set, so HF reports a clear error at download time. Adds a regression test. * Studio: strip blank HF_HOME and isolate cache-env tests Address review: a whitespace-only HF_HOME no longer derives " /hub"; strip it and fall back to the default (matches studio_root). Tests set UNSLOTH_STUDIO_HOME to a tmp dir so _setup_cache_env's UV/VLLM mkdirs do not touch the real ~/.unsloth/studio. Adds a whitespace regression test. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-06-22 08:21:59 -07:00
Daniel Han	368b19b237	Studio: fix training output dir escaping outputs root for models on another drive (#6293) ... * Studio: derive training output dir from model basename for local-drive models A LoRA/QLoRA run started from a model loaded by absolute path (common when models live on a non-system drive, e.g. G:\modelsAI\...\gemma-4-12B-it) seeded the default output dir with that full path. resolve_output_dir then raised "path escapes root ... is not under the studio outputs folder", so training could not start from a model stored off the system drive. Add default_run_dir_name(): Hugging Face repo ids keep their namespace (org/model becomes org_model), while local paths collapse to their final component so an absolute source path can no longer leak into the output dir. Use it at the three worker derivation sites and add a regression test. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: cap run dir name length and drop redundant resolve Apply PR review feedback: length-cap the auto-generated output dir component so an unusually long model name stays under the filesystem name limit, and drop the redundant double resolve_output_dir at the embedding site so all three derivation sites match. --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-06-13 04:06:17 -07:00
Anmol Mishra	554c289538	fix: respect absolute export paths to prevent cross-drive copy failures (WinError 112) (#6088) ... * fix: allow absolute save_directory in export paths to prevent cross-drive copy failures The GGUF export pipeline (and all other export flows) forced every save_directory through resolve_export_dir(), which always resolved the path under exports_root() — typically ~/.unsloth/studio/exports/ on the system drive (C: on Windows). When a user selected an output directory on a different drive (E:): 1. The absolute path was rejected at the Pydantic validator level. 2. Even if it got through, resolve_export_dir would re-resolve it under C:\Users\.unsloth\studio\exports\. 3. After GGUF conversion completed on E:, the relocation step would try to move/copy the finished files to C:, causing: - WinError 17 (cross-drive move failure when shutil.move falls through to a cross-filesystem copy) - WinError 112 (disk full on C:) Fix both layers: - _validate_save_directory: accept absolute paths (they represent an explicit user choice of output location). - resolve_export_dir, resolve_output_dir, resolve_tensorboard_dir: return absolute paths as-is instead of forcing them under the default root. Keep the existing safety checks (null bytes, '..' segments) and fall through to resolve_under_root for relative paths. Fixes: https://github.com/unslothai/unsloth/issues/6082 * refactor: centralize user path validation into _resolve_user_path helper Addresses code review feedback: the null-byte, '..', and absolute-path checks were duplicated across resolve_output_dir, resolve_export_dir, and resolve_tensorboard_dir. Extract a single _resolve_user_path helper that all three delegate to. No behavioral change — pure consolidation. * fix: address code review — contain destructive cleanup and scope absolute paths Address all review feedback from gemini-code-assist: 1. P1: destructive subdirectory cleanup (export_gguf) The flattening loop in export_gguf previously rmtree'd every subdirectory under abs_save_dir. When targeting an existing user directory on a different drive (#6082), this could nuke unrelated subdirectories. Now snapshot existing subdirectories before the export and only clean up dirs created during this run. 2. P2: keep scan/read endpoints contained Only resolve_export_dir accepts absolute paths (export is a write path where user picks location). Reverted resolve_output_dir and resolve_tensorboard_dir to use resolve_under_root directly — these are used by scan/read/training endpoints that must stay contained under their respective roots. 3. Centralization feedback Removed the _resolve_user_path helper since it's no longer needed with the narrowed scope. resolve_export_dir has the absolute path logic inline with a clear docstring. * fix: skip pre-existing subdirs in GGUF flatten loop and clean stale export intermediates Two issues caught in code review (chatgpt-codex-connector): 1. The flattening loop moved ALL .gguf files from ALL subdirectories into abs_save_dir, including pre-existing unrelated user subdirs. Now skip pre-existing subdirs entirely unless they are known export-owned intermediates (model/, model_gguf/). 2. After a failed export, known export-owned subdirectories (model/, model_gguf/) were snapshotted as pre-existing on retry and never cleaned up. These are now always cleaned up regardless, since they are known intermediates created by the export pipeline. * fix: separate write vs read export paths, guard same-dir rmtree Three issues caught in code review (chatgpt-codex-connector): 1. P1: scan endpoint containment resolve_export_dir was changed to accept absolute paths, but it's also used by scan/read endpoints (routes/models.py) that must stay contained under exports_root(). Split into: - resolve_export_dir: contained, used by scans - resolve_export_write_dir: accepts absolute paths, used by export backend only 2. P1: same-directory rmtree When a non-PEFT checkpoint's gguf_dir resolves to the same path as abs_save_dir (user selected the checkpoint's gguf output as their export directory), shutil.rmtree(gguf_dir) would delete the user's chosen output directory. Now skip relocation when both paths resolve to the same location. 3. P1: pre-existing subdir flatten loop Reverted _EXPORT_OWNED_SUBDIRS logic — 'model/' and 'model_gguf/' are common directory names in shared model folders and don't prove export ownership. Now only clean up subdirs that didn't exist before the export started. * fix: remove dead _EXPORT_OWNED_SUBDIRS and fix _export_details for absolute paths Two fixes from review comments: 1. Remove unused _EXPORT_OWNED_SUBDIRS declaration (leftover from previous iteration that was intentionally removed). 2. _export_details now returns the full absolute path when the export target is outside exports_root(), instead of truncating to basename. Users who export to E:\ can now see the full destination path in the success dialog. * fix: use unique tmp dir for GGUF intermediates to avoid overwriting user dirs When exporting to an absolute destination that already contains a model/ subdirectory (e.g. a shared models folder), the hard-coded model_save_path would overwrite files in that unrelated directory. Use _tmp_model_<uuid> as the intermediate path instead, so user directories are never touched. The tmp dir is created as a new subdir of abs_save_dir and cleaned up by the flatten loop after GGUF files are relocated. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix GGUF local export paths for PR #6088 * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Address GGUF export follow-ups for PR #6088 * Clean GGUF temp dirs on export failure for PR #6088 * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix/adjust export path tests for PR #6088 * Fix/adjust export path review findings for PR #6088 * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix/adjust home export path handling for PR #6088 --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: wasimysaid <wasimysdev@gmail.com>	2026-06-12 12:52:57 +02:00
Daniel Han	1a99980b46	Studio: auto Cloudflare tunnel for 0.0.0.0 launches (#6204) ... * Studio: auto Cloudflare tunnel for 0.0.0.0 launches Binding Studio to 0.0.0.0 for remote access often leaves the raw http://<ip>:<port> URL unreachable (https-vs-http, blocked high ports, closed cloud security groups). On a wildcard bind, auto-start a free cloudflared quick tunnel and show its https://*.trycloudflare.com URL in the startup banner: Secure link access via Cloudflare: https://<random>.trycloudflare.com - new studio/backend/cloudflare_tunnel.py: find or download+cache the cloudflared binary (per-OS/arch GitHub release, safe .tgz extract), start the tunnel, parse the URL, tear it down. Stdlib only; best-effort and non-fatal throughout (a missing binary or offline box never blocks or slows startup). - run_server starts the tunnel for 0.0.0.0 only (skips loopback, api-only and Colab), prints the line in the banner, and _graceful_shutdown stops the child so it never orphans. - --cloudflare/--no-cloudflare flag (default on) on `unsloth studio` and `unsloth studio run`, forwarded through the re-exec into run_server. - tests for the helper, the CLI flag forwarding, and the run.py defaults. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio cloudflare: send a User-Agent on the cloudflared download GitHub's CDN can 403 the default Python-urllib User-Agent on release asset downloads. Set an explicit UA and pin it with a test. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio cloudflare: address review (opt-out for subcommands, tunnel teardown) - reject --no-cloudflare placed before a subcommand (it would not reach the subcommand), mirroring the --parallel guard - register the tunnel before waiting for its URL so a shutdown during the wait stops cloudflared instead of orphaning it - tear the server + children down if `unsloth studio run` startup aborts (health timeout, model-load error, Ctrl+C) before the wait loop * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-06-11 07:10:08 -07:00
Daniel Han	8848a310df	Studio: clean-room compact RAG (knowledge bases, hybrid search, fast indexing) (#5910) ... Adds a self-contained RAG stack to Studio: knowledge bases with chunked indexing, hybrid (dense + lexical) retrieval, and an automatic first-pass context inject into chat. Embeddings run through a local llama-server GGUF backend (default unsloth/bge-small-en-v1.5-GGUF) with a sentence-transformers fallback. The chat tool loop gains a search_knowledge_base tool, a per-turn re-search cap, and source citation, layered on top of the shared ToolLoopController.	2026-06-09 21:17:04 -07:00
Daniel Han	187144d4e7	Reduce and tighten code comments and docstrings repo-wide (#6095) ... Trim and tighten code comments and docstrings across the repository. Comment-only: every changed file verified code-identical to main via AST/token comparison.	2026-06-08 23:09:51 -07:00
Daniel Han	8292e699e4	Studio: make code comments and docstrings more succinct (#6029) ... Trim and tighten code comments and docstrings across studio/ Python. Comment-only: every changed file verified code-identical to main via AST/token comparison.	2026-06-08 23:07:28 -07:00
Daniel Han	3ce187da02	Formatting: ruff line-length 100, kwarg-spacing passes, drop blank after short local imports (#6079) ... Raise ruff line-length to 100 and extend the local pre-commit format pipeline (def-signature magic-comma normalization, short multi-line assert collapse, kwarg '=' spacing, blank-line-after-short-import removal, adjacent string-literal / f-string+plain merge, redundant-pass pruning). Every transform re-checks the file AST and is dropped if it would differ; the whole-repo reformat is verified AST-identical per file and idempotent.	2026-06-08 04:24:13 -07:00
Lee Jackson	e0ff6a1404	Studio: manage chat history with projects (#5725) ... * feat: align project sidebar UX with ChatGPT * feat: align project sidebar UX with ChatGPT * feat(chat): load stored project list * feat(chat): add project sidebar workflows * fix: stabilize project page navigation * fix: projects chat loading * fix: show project chat thread * style: sidebar project spacing and hover clipping * style: add expandable project chat history and move-to-project submenu * feat: polish project sidebar * feat: persist project sandbox paths * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix: only create sandbox project workspace dir * feat: add optional project workspace deletion from delete dialog * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix: stabilize chat projects CI failures * fix: polish project chat navigation * Studio: manage chat history with projects Group chats into projects with a dedicated projects page and route. Sidebar shows recents with per-row actions and a vertical more-vertical menu, and the sidebar scrollbar stays hidden so rows never shift on hover. Includes chat settings and composer refinements. * Studio: projects sidebar and breadcrumb polish Sidebar: - Remove the Compare nav item. - Widen the sidebar to match the projects layout. - Replace the scroll-gated bottom fade with a static fade pinned above the profile box, so it no longer attaches to Recents or lags the collapse and expand animation. Topbar breadcrumb (chat-page): - On a project landing show "Projects" linking to the projects list. - Inside a project chat show the project name and chat title, with the project name linking back to that specific project page. - Drop the divider between the model selector and the breadcrumb. * Studio: make project workspace delete test cross-platform test_chat_project_delete_files_removes_workspace rooted the project under pytest tmp_path, which resolves to /private/tmp on macOS. The workspace delete guard refuses paths under the system denylist by design, so the test passed on Linux CI but failed on macOS. Add a workspace_projects_home fixture that keeps tmp_path on Linux and Windows (CI unchanged) and falls back to a home subdir only when the temp root is on the platform denylist. Derive the workspace path from the created project so it tracks the projects home. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: satisfy import-hoist check for new path re-exports documents_root and project_workspaces_root are re-exported from utils.paths but only referenced as __all__ string literals, which the import-hoist safety net does not count as a use. It flagged the two newly added re-exports as unused imports and failed Source lint. Name-load both via a module-level _REEXPORTED tuple so the check sees them used. No behaviour change; consumers still import them from utils.paths. * fix: avoid projects empty-state flash * fix: batch chat search indexing * Studio: polish chat sidebar, run settings, and search - Use the native OS scrollbar for the chat sidebar, Run settings panel, and chat search list instead of a custom scrollbar - Highlight the active run in the sidebar and keep chat search available during training - Stop the training log view from replaying when navigating back to a run - Rename the chat settings panel to Run settings and align its toggle icon and position - Tighten heading and sidebar letter spacing and lighten the Train and Recents labels - Match the search dialog corner style across light and dark and drop the stray border - Make the MCP Servers section header plain text instead of a link - Remove a stray .orig backup file * studio/frontend: restore Compare entry point in the sidebar The chat-projects sidebar redesign dropped the Compare nav item and moved it to thread-sidebar.tsx, which is not imported or rendered anywhere. That left no way for a user to start a new model comparison (enterCompare only fired from the guided tour and the training handoff), and broke the Compare/Recipes/Export UI smoke test that clicks [data-tour="chat-compare"]. Re-add the Compare NavItem to the New Chat / Search group, carrying data-tour="chat-compare" and the same new-comparison navigation as before. * studio/frontend: use Unsloth green for the fallback profile avatar Switch the initials-avatar background from blue to #14b789 so the sidebar and edit-profile avatar match the Unsloth brand colour. * studio/frontend: turn project breadcrumb into a project switcher dropdown * studio/frontend: stop project card kebab clicks from opening the project * studio/frontend: hide project switcher outside projects * studio/frontend: stabilize project switcher loading * style: project switcher alignment --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: shimmyshimmer <107991372+shimmyshimmer@users.noreply.github.com> Co-authored-by: Unsloth <michaelhan@Michaels-MacBook-Pro.local> Co-authored-by: Roland Tannous <rolandtannous@gravityq.ai> Co-authored-by: Roland Tannous <115670425+rolandtannous@users.noreply.github.com>	2026-06-01 22:09:16 +04:00
Daniel Han	0881a7a5d7	studio: security and hardening pass (auth rate-limit, sandbox, path containment, schema validation, headers) (#5375) ... * studio: contain export and dataset paths under their configured roots resolve_under_root and resolve_dataset_path previously returned absolute paths unchanged, so an authenticated client could supply save_directory="/tmp/escape" (or any other absolute path) and have the exporter drop adapter files anywhere the server user could write. This turned up during a recent audit pass where an authenticated POST to /api/export/export/lora with save_directory="/tmp/lora_escape_test" returned 200 and wrote adapter_model.safetensors, adapter_config.json, and tokenizer files under /tmp. The fix is two-layered: storage_roots.py adds an _assert_contained(resolved, root) helper that runs after path resolution and rejects any result whose realpath does not sit under realpath(root). resolve_under_root now rejects '..' segments and null bytes outright, and only accepts absolute inputs when they are already inside the configured root (internal call sites that re-resolve a stored absolute path stay idempotent; worker.py:resolve_output_dir(output_dir) etc. continue to work). resolve_dataset_path picks up the same containment rule, scoped to the three dataset roots. models/export.py adds field_validator("save_directory", mode="before") to ExportCommonOptions and ExportGGUFRequest so bad input fails fast at 422 with a clear message rather than a 500 deep inside the resolver. The validator rejects empty/whitespace, null bytes, control chars, strings longer than 255 chars, absolute paths, and '..' segments. routes/export.py:_export_details now returns os.path.relpath(output_path, exports_root()) so the Export Complete dialog and /api/models/loras no longer leak the absolute install prefix to the UI; the basename is used as a last-resort fallback. Verified end to end: - POST /api/export/export/lora {"save_directory":"/tmp/foo"} -> 422 "save_directory must be a name or relative path under the export root; absolute paths are rejected". /tmp/foo is not created. - "../../etc/escape" -> 422 "may not contain '..' segments". - save_directory="my_subdir" -> still accepted (400 only because the test had no checkpoint loaded yet, not because of validation). - Internal idempotent re-resolve via resolve_export_dir(absolute path that is already under exports_root) returns the same path unchanged. * studio/sandbox: harden bash + python tool execution The sandboxed Bash and Python tool channels in Chat ran with a thin preexec hook (PR_SET_NO_NEW_PRIVS + RLIMIT_FSIZE only). Bash had a small word blocklist; Python had an AST safety pass aimed at signal-tampering and shell-escape primitives. An audit pass showed several gaps that a tool-calling model could trigger inadvertently: - bash curl/wget/nc reached AWS IMDSv2 and returned live STS credentials for the instance role. - python "import socket; s.connect((169.254.169.254, 80))" reached the same endpoint regardless of the bash blocklist. - "cat /etc/passwd" was blocked at the bash side (because "passwd" is in the blocklist), but "open('/etc/passwd').read()" in Python happily returned its contents. - "chr(115)+chr(117)+chr(100)+chr(111)" style dynamic-arg construction slipped through the AST shell-escape check. - The supervisor used proc.kill() on timeout, which only signals the immediate pid; bash-backgrounded children survived. A fork bomb could spawn for the full 300s timeout window. - Session work directories under ~/studio_sandbox/<id>/ were created with default umask (0o755), so any other UID on the host could enumerate them. - session_id sanitisation used a one-shot str.replace("..",""), which is non-iterative and a small footgun. This commit takes a conservative middle path: the sandbox still runs as the Studio UID with no namespace tricks where the kernel disallows them, but every chokepoint is tightened. _sandbox_preexec now: - calls os.setsid() so children share a process group; the supervisor uses os.killpg(SIGKILL) on timeout/cancel so backgrounded children die with the parent (new _kill_process_tree helper, wired into _cancel_watcher and both _bash_exec / _python_exec timeout branches). - calls os.umask(0o077) so files the child writes default to 0o600. - applies PR_SET_PDEATHSIG=SIGKILL so an orphaned child dies if Studio exits. - best-effort unshare(CLONE_NEWNET) for a private network namespace (failure is logged and swallowed; defense-in-depth is still in place via the bash blocklist and the AST checker below). - sets RLIMIT_NPROC=10000 (tunable via UNSLOTH_STUDIO_SANDBOX_NPROC), RLIMIT_AS=8GB, RLIMIT_CPU=300, RLIMIT_NOFILE=1024. The 10k NPROC figure is chosen to sit well above the ~500 LWPs a healthy Studio + llama-server combination already uses while still capping a runaway fork bomb. NPROC counts LWPs per real UID, so a lower figure (e.g. 256) starves legitimate bash forks ("bash: fork: retry: Resource temporarily unavailable"). _get_workdir: - rejects session_id that doesn't match [A-Za-z0-9_-]{1,64}; non-matching values bucket into a shared "_invalid" dir. - chmod 0o700 on both the workdir and on ~/studio_sandbox/ so other UIDs cannot read another session's contents. _BLOCKED_COMMANDS_COMMON gains: doas, pkexec, halt, poweroff, curl, wget, nc, ncat, netcat, socat, ssh, scp, sftp, rsync, eval, source. The intent is to keep general bash usage working (echo, ls, pipes, loops, for, head, etc.) while denying the obvious egress and escalation paths. The AST checker (_check_signal_escape_patterns) is split into the existing shell/signal/loop checks plus a new narrow IO denylist: - Always flag non-literal args to anything in _SHELL_EXEC_FUNCS, not just _STRING_SHELL_FUNCS. Closes the dynamic-arg bypass. - Reject calls to socket.create_connection, socket.socket().connect, urllib.request.urlopen, http.client.HTTP*Connection, requests.*, httpx.* whose literal host argument is in a cloud-metadata denylist (169.254.169.254 + 169.254.* + 100.64.*, plus the GCP/Alibaba/ECS metadata hostnames and IPv6 link-local). Public hosts (example.com, huggingface.co, ...) still work. Dynamic hosts cannot be statically blocked; mitigated by the bash blocklist + the netns where the kernel allows it. - Reject literal open("/etc/passwd"), /etc/shadow, /etc/sudoers, /etc/ssh/*, and /proc/<pid>/environ. Other files (/etc/os-release, /etc/hostname, /tmp/*, user dirs) still work. The _check_code_safety summariser is updated to include the new network_calls and sensitive_file_reads buckets in its error string. Regression-checked: echo, sleep, ls /tmp, for loops, piped helpers (echo a | tr a A), urllib.request.urlopen("http://example.com"), socket.getaddrinfo("example.com",80), open("/etc/os-release"), open("/tmp/...","w") all still succeed. curl, wget, nc, ssh, rm, socket.create_connection(("169.254.169.254",80)), open("/etc/passwd"), open("/proc/self/environ") all correctly blocked. * studio: rate-limit login, rotate refresh tokens, add logout, security headers, gate bootstrap injection A pass over the auth surface found a cluster of related issues that this commit closes together. Login (routes/auth.py): - Add an in-memory per-IP login rate limiter. Five failed POSTs to /api/auth/login inside a 60s window produce 429 with Retry-After. A successful login clears the bucket. Previously 30 wrong passwords in under one second was accepted as 30x 401, which combined with the (now fixed) admin-username leak from /api/auth/status made brute-force trivial against a small password. Logout (routes/auth.py): - New POST /api/auth/logout returns 204 and calls storage.revoke_user_refresh_tokens(subject) so the refresh token is no longer valid. Previously POST /api/auth/logout returned 405 and there was no way to invalidate refresh tokens short of changing the password. Frontend session.ts already calls clearAuthTokens() to drop localStorage; the new endpoint lets the client also tell the server to revoke server-side state. Refresh-token rotation (routes/auth.py + auth/storage.py): - New storage.consume_refresh_token(token) atomically validates + deletes a refresh token, returning (username, is_desktop). The /api/auth/refresh handler now mints both a new access AND a new refresh token; the supplied token becomes invalid. Replaying a consumed refresh returns 401 "Invalid or expired refresh token". The previous refresh_access_token helper is left in place for callers that intentionally want the non-rotating shape; nothing in the route layer uses it now. /api/auth/status no longer leaks default_username (models/auth.py + routes/auth.py): - AuthStatusResponse.default_username becomes Optional[str] with a None default; the handler always returns None. The frontend already hardcodes HIDDEN_LOGIN_USERNAME = "unsloth" (auth-form.tsx:82), so no UI change is required. window.__UNSLOTH_BOOTSTRAP__ no longer auto-injects (main.py): - _inject_bootstrap is now opt-in via the UNSLOTH_STUDIO_INJECT_BOOTSTRAP env var. The previous default (inject whenever requires_password_change is true) embedded the plaintext bootstrap password into the first-boot HTML for any caller that hit /, /change-password, or any unknown SPA path. Browser extensions and any XSS payload on the page could read it trivially. With the new gate the bootstrap password lives only in the auth/.bootstrap_password file (mode 0o600) where it has always been; users typing it into a current-password field is the right UX. routes/auth.py:change_password also clears app.state.bootstrap_password defensively. Security headers + server fingerprint (main.py + run.py): - New SecurityHeadersMiddleware adds Content-Security-Policy, X-Frame-Options: DENY, X-Content-Type-Options: nosniff, Referrer-Policy: no-referrer, Permissions-Policy: camera=(), microphone=(), geolocation=(), interest-cohort=(), and stamps server: unsloth-studio so the generic uvicorn banner no longer fingerprints the stack. The uvicorn.Config gains server_header=False so it stops emitting its own Server header. /api/health minimisation (main.py): - Unauthenticated GET /api/health returns just {"status":"healthy","timestamp":...} so load-balancer liveness probes keep working without leaking version, device_type, chat_only, desktop_protocol_version, or studio_root_id to arbitrary callers. A request that presents a valid Bearer token still gets the full diagnostic payload so internal launchers and sibling-Studio detection (which compares studio_root_id) keep working. Verification: - 30 wrong-password POSTs to /api/auth/login -> first 5 = 401, 6th through 30th = 429. - POST /api/auth/logout with a fresh token -> 204. The matching refresh token then fails 401. - Login -> R1; /api/auth/refresh with R1 -> new access + R2 (R2 != R1); /api/auth/refresh with R1 again -> 401; /api/auth/refresh with R2 -> still succeeds once and rotates again. - curl /api/auth/status -> default_username: null. - curl http://127.0.0.1/ does not contain __UNSLOTH_BOOTSTRAP__. - curl -I / shows CSP, X-Frame-Options: DENY, X-Content-Type-Options: nosniff, Referrer-Policy: no-referrer, Permissions-Policy, and server: unsloth-studio. - curl /api/health unauthenticated -> {status, timestamp} only. curl with Authorization: Bearer <valid> -> full payload. - Existing /api/system, /api/models/list, /api/train/status, /api/inference/status, /api/auth/api-keys, login flow, SPA root all still return 200 after the changes (regression smoke). * studio: add SecurityHeadersMiddleware, MaxBodyMiddleware, /recipes redirect, gate _inject_bootstrap, minimise /api/health This commit lands the main.py-side changes that share a single middleware-registration spot. They are kept together because every change here is either (a) a top-level middleware definition that has to be added next to LoggingMiddleware, or (b) a route handler at the same file-level. SecurityHeadersMiddleware (Content-Security-Policy, X-Frame-Options: DENY, X-Content-Type-Options: nosniff, Referrer-Policy: no-referrer, Permissions-Policy, server: unsloth-studio). The previous responses emitted no CSP, no XFO, no Referrer-Policy and were stamped server: uvicorn. MaxBodyMiddleware rejects POST/PUT/PATCH on the inference / dataset / data-recipe / train / export prefixes when Content-Length exceeds UNSLOTH_STUDIO_MAX_BODY_MB (default 100). The audit hit this by attaching a 50 MB plain-text file to a chat message and watching Studio base64-encode it into the JSON body; uvicorn has no enforced cap so the only previous guard was the per-file 50 MB ceiling that data-recipe upload routes already enforce. The new middleware extends that ceiling to the OpenAI-compat path that the Chat attachments flow through. Verified: a 200 MB JSON POST to /v1/chat/completions returns HTTP 413 "Request body too large (209,715,264 bytes; max 104,857,600)". A small valid request continues to reach the handler. _inject_bootstrap is gated behind UNSLOTH_STUDIO_INJECT_BOOTSTRAP. The previous default was to inline window.__UNSLOTH_BOOTSTRAP__ = {username, password} into the first-boot HTML whenever requires_password_change was true, which exposed the plaintext bootstrap password to any browser extension, page script, or LAN caller on -H 0.0.0.0. The bootstrap password remains in the on-disk .bootstrap_password file (mode 0o600) where it has always lived; users typing it into a current-password field is the right UX. /api/health unauthenticated returns {"status":"healthy","timestamp": ...} only; the previous payload (version, device_type, chat_only, desktop_protocol_version, supports_desktop_auth, studio_root_id, native_path_leases_supported) is preserved for callers that present a valid Bearer token, so internal launchers and sibling-Studio detection (which compares studio_root_id) keep working. /recipes -> /data-recipes 308 redirect. The Data Recipes page lives at /data-recipes; users typing /recipes hit the SPA catch-all and saw "Not Found". The redirect also preserves any tail path, so /recipes/<rest> -> /data-recipes/<rest>. Verified end to end with curl: CSP / XFO / X-Content-Type-Options / Referrer-Policy / Permissions-Policy all present on /, server header is now unsloth-studio (uvicorn's own banner is suppressed via server_header=False in run.py from the auth-batch commit). Followed the /recipes redirect lands on the SPA HTML. * studio: bound TrainingStartRequest hyperparameters at the schema level POST /api/train/start accepted any value for learning_rate, batch_size, max_steps, max_seq_length, warmup_steps, warmup_ratio, num_epochs, save_steps, weight_decay, gradient_accumulation_steps, lora_r, lora_alpha and lora_dropout, including -1, 0, 1e9, and non-numeric strings like 'abc' or 'two' (which silently coerce to 0 in the trainer). Probing showed the API returning 200 to learning_rate=-1 and batch_size=0; only max_steps had any partial clamping. This commit adds field_validator on every numeric hyperparameter. Bounds are chosen wide enough to span realistic single-host configurations (B200 with 180 GB of memory comfortably fits the upper end) while rejecting the values that always produce broken training: - learning_rate: parses str/float, requires 0 < lr < 1.0. Non-numeric input raises with "learning_rate must be parseable as float (got 'abc')" instead of silently coercing to 0. - batch_size: [1, 1024]. - gradient_accumulation_steps: [1, 4096]. - num_epochs: [1, 1000]. - max_steps: [1, 1_000_000]. - max_seq_length: [1, 131072]. - warmup_steps: [0, max_steps]. - warmup_ratio: [0.0, 1.0]. - save_steps: [0, 1_000_000]. - weight_decay: [0, 10] (typical 0..0.1). - lora_r: [1, 512]. - lora_alpha: [1, 1024]. - lora_dropout: [0.0, 1.0). Each validator names the offending field in its ValueError message so the 422 response body identifies which input is bad. The learning_rate validator returns its result as str (the schema field type is str("2e-4") for backwards compatibility) so existing call sites that float() the value continue to work. Verified: - learning_rate=-1 -> 422 "learning_rate must be > 0 (got -1.0); typical range is 1e-6 .. 1e-3". - learning_rate='abc' -> 422 "must be parseable as float". - batch_size=-1 / 0 / 999999 -> 422 "batch_size must be in [1, 1024]". - batch_size='two' -> 422 (pydantic int parser). - max_steps=0 / -5 -> 422 "must be a positive int". - max_seq_length=200000 -> 422 "must be in [1, 131072]". - warmup_ratio=2.5 -> 422 "must be in [0.0, 1.0]". - lora_dropout=1.5 -> 422 "must be in [0.0, 1.0)". - Valid request with learning_rate='2e-4', batch_size=1, max_steps=5 passes validation and the training run starts as normal. * studio: redact image-decode errors, clean checkpoint dirs on cancel, tolerate Stop-button + tool-result message shapes Three small fixes that fall under "do not let the audit findings become user-visible papercuts". routes/inference.py - image-decode error redaction (the audit hit this with a 0-byte / malformed / wrong-extension image upload). The three image-normalise sites previously raised HTTPException(400, detail=f"Failed to process image: {e}"). When PIL raised UnidentifiedImageError(io.BytesIO(raw)) the message string included "<_io.BytesIO object at 0x7e40a5d7bf60>", leaking both the Python class name (confirming the PIL/io stack) and a heap address (mildly useful for ASLR-bypass chaining if another memory-corruption bug is ever found). Each site now catches UnidentifiedImageError and returns the generic "Unsupported or corrupt image format"; the fall-through generic except returns "Failed to process image". No exception-repr is interpolated into a response body anywhere along these paths. core/training/training.py - checkpoint cleanup on cancel. When a user clicks Cancel Training, the trainer flips _cancel_requested=True and the supervisor force-terminates the subprocess. The trainer writes checkpoint-<step> directories under output_dir every save_steps; previously these survived the cancel and accumulated on disk (the audit recorded ~67 MB stuck after a 200-step cancel with save_steps=20). New helper _cleanup_cancelled_checkpoints(output_dir) globs checkpoint-<int> entries and removes them. It is gated by a realpath containment check against outputs_root() so it cannot accidentally rmtree anything outside the configured outputs root. force_terminate() invokes the helper after the subprocess join when _cancel_requested is true. Stop-and-Save runs are unaffected because that path keeps _cancel_requested=False. models/inference.py - chat message shape tolerance. Two related frontend interactions used to crash the request validator: - After the Stop button truncates a generation, the frontend retained {role:"assistant", content:""} in the conversation history and replayed it on the next send. ChatMessage previously required role="assistant" to have non-empty content or tool_calls, so the next message returned 422 and the thread was permanently broken. The validator now normalises empty assistant content to None so the request round-trips and the trailing empty turn can be ignored downstream. - The frontend's second-round tool POST drops the streamed tool_call_id, hitting the strict-spec check "role=tool requires tool_call_id". The validator now synthesises an opaque id (call_<8 hex>) when missing, so the request reaches the handler and the model's final summarising response gets generated. The proper fix lives in the frontend (carry the streamed id through the second POST) and will follow. Verified end to end with curl: HTTP 400 (model not loaded) on both the empty-assistant history shape and the tool-result-without-id shape, instead of HTTP 422 from the schema validator. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * studio: tighten code comments from security-hardening pass Trim verbose docstrings and inline finding references added in the previous commits in this branch. Functionality unchanged. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * studio: await get_current_subject in /api/health and make refresh-token consumption atomic The /api/health auth probe called get_current_subject(creds) without awaiting it. The coroutine object is truthy, so any caller presenting a Bearer header (valid or not) received the full diagnostic payload including version, device_type, studio_root_id, etc. Await the coroutine and treat HTTPException as 'fall back to the minimal liveness payload'. consume_refresh_token did SELECT then DELETE WHERE id under default autocommit isolation. Two concurrent POST /api/auth/refresh requests could both win the SELECT before either DELETE ran, defeating single-use refresh-token rotation. Replace with a single DELETE ... WHERE token_hash = ? AND expires_at >= ? RETURNING ... statement so the validate-and-delete lands as one atomic op under SQLite's write lock (3.45.1 supports RETURNING; min was 3.35). * studio: enforce body cap on chunked uploads and drop unsafe-inline from script-src MaxBodyMiddleware previously only inspected the declared Content-Length header; clients omitting it or sending Transfer-Encoding: chunked bypassed the cap and could still drive an OOM via the downstream JSON / file readers on /v1/chat/completions, /api/inference, /api/data-recipe, /api/datasets, /api/train, /api/export. Rewrite as a raw ASGI middleware that drains and counts http.request frames, replies 413 once the running total exceeds UNSLOTH_STUDIO_MAX_BODY_MB before invoking the FastAPI handler, and replays the buffered body to downstream so route code that calls request.json() / await request.body() works unchanged. CSP previously included 'unsafe-inline' on script-src, which defeats the main XSS protection. The frontend bundle does not need inline scripts; the only inline <script> the backend ever emits is _inject_bootstrap, which is opt-in via UNSLOTH_STUDIO_INJECT_BOOTSTRAP. Drop 'unsafe-inline' from script-src by default; when _inject_bootstrap fires, generate a per-response nonce, embed it on the inlined <script>, and have SecurityHeadersMiddleware splice 'nonce-XXX' into the CSP for that one response (the internal x-internal-script-nonce header is popped before the response leaves the server). 'unsafe-inline' stays on style-src for Vite-injected styles. * studio: drop empty assistant sentinel before passthrough ChatMessage._validate_role_shape normalises role="assistant", content="" (the post-Stop sentinel emitted by the frontend) to content=None so the in-process path can drop it via _extract_content_parts. The passthrough path then ran m.model_dump(exclude_none=True), which strips the now-None content key entirely, sending {"role":"assistant"} to llama-server / the OpenAI-compat backend. That fails upstream and leaves the user without a recoverable Stop->resume. Add _drop_empty_assistant_sentinels and call it at both passthrough message origins: _openai_messages_for_passthrough (covers /v1/chat/completions and the Responses API which routes through it) and the anthropic_messages_to_openai output before _anthropic_passthrough_*. Assistant messages that carry only tool_calls (no content) are preserved. * studio/tests: cover audit-fix surfaces and rebase pre-existing tests Adds and updates pytest coverage for the four bot-flagged audit fixes landed earlier in this branch and rebases two pre-existing tests that were broken by the relaxed-validator and /api/health auth-gate changes. studio/backend/tests/test_middleware.py (new) MaxBodyMiddleware: small protected, large declared, unprotected passthrough, chunked-upload-over-cap rejection (the regression for the original Content-Length-only gap), and chunked-under-cap replay. SecurityHeadersMiddleware: script-src no longer carries 'unsafe-inline', style-src still does, default headers (XFO/XCTO/Referrer-Policy/Permissions-Policy/server), and the internal x-internal-script-nonce header is consumed by the middleware and converted to 'nonce-XXX' in the CSP. /api/health: no auth -> minimal, invalid Bearer -> minimal (the await regression), valid Bearer -> full diagnostic payload. studio/backend/tests/test_desktop_auth.py consume_refresh_token: second-call returns None, expired returns None, and a 64-thread concurrent pile-up against the same hash produces exactly one successful consumer (regression for the SELECT-then-DELETE race). test_health_response_reports_desktop_capability_fields: rebase against the new health_check(request) signature by going through TestClient with a real bearer instead of asyncio.run-ing the handler directly. studio/backend/tests/test_openai_tool_passthrough.py Pin the new ChatMessage tolerance: assistant without content or tool_calls is tolerated (normalises content -> None), empty-string and empty-list assistant content normalise to None, and a missing / empty tool_call_id on role='tool' is synthesised as call_<hex> rather than raising. Tests for _drop_empty_assistant_sentinels cover the three drop shapes (empty string, empty list, missing content key), preservation of assistant text and tool_calls-only messages, and end-to-end through _openai_messages_for_passthrough. studio/backend/main.py SecurityHeadersMiddleware.dispatch used response.headers.pop(...) for the nonce-header handoff; Starlette's MutableHeaders has no pop. Read-then-del so the internal handoff header is still stripped before the response leaves the server. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * studio/tests: rebase three more pre-existing CI tests against this branch CI on PR #5375 was red on three tests that were tuned for behaviour predating this branch. Updates each so the assertions match what the audit fixes intentionally changed; no production code touched. studio/backend/tests/test_trained_model_scan.py test_scan_trained_models_includes_lora_and_full_finetune_outputs passed an absolute tmp_path through scan_trained_models, which now runs resolve_output_dir / _assert_contained against outputs_root(). Repoint outputs_root() at tmp_path via monkeypatch so the fixture dirs land under the configured root and the realpath containment check passes. tests/test_studio_install_workspace_guard.py test_health_endpoint_exposes_studio_root_id_not_raw_path read the first 1500 bytes after @app.get("/api/health") and asserted on the studio_root_id literal. The handler grew (unauth short-circuit + await dependency gate) and the literal slid past the byte window. Replace the fixed window with a slice up to the next top-level @app.* decorator so the test surveys the whole handler regardless of size. tests/studio/studio_api_smoke.py The "login burst (5x wrong pw) -> 401 each" assertion was tagged "When/if we add one, this assertion updates in the same PR." We added the per-IP rate-limit in routes/auth.py (_LOGIN_MAX_FAILS=5/60s) but missed the assertion update. Rewrite the burst probe to observe the new invariant: at least one 401, eventual transition to 429, and Retry-After present on the 429. Adds a small _login_with_headers helper since the existing login() helper drops response headers. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * ci(studio-ui): set UNSLOTH_STUDIO_INJECT_BOOTSTRAP=1 for Playwright Studios The Chat UI Playwright test drives the first-boot change-password form, which (per playwright_chat_ui.py step "1. Change-password through the UI") pre-seeds the hidden current_password field from window.__UNSLOTH_BOOTSTRAP__. That global is only emitted when the backend's _inject_bootstrap path fires, which since the security pass on this branch is gated behind UNSLOTH_STUDIO_INJECT_BOOTSTRAP and defaults to off. Without the global, the React form's current_password validator never satisfies, the submit button stays disabled, and the composer.wait_for() probe times out on /change-password. Re-enable injection only for the CI Studios that drive the chat UI across linux/mac/windows. Production deployments are unaffected: the env var has to be explicitly opted into, and the on-disk auth/.bootstrap_password remains the source of truth for human users typing the password in by hand. Covers all eight Studio launch sites: the primary chat-ui boot and the "extra UI tests" boot for each of the three OSes, plus the pipeTransport JSON-crash retry relaunches in the macOS workflow that re-spawn Studio mid-job. A follow-up frontend PR will add a visible current_password input so the form satisfies its own validator without needing the bootstrap auto-fill at all; once that lands this CI knob can come back out. * studio/sandbox: drop unshare(CLONE_NEWNET); add trusted-host allowlist; block sandbox file uploads; raise CPU rlimit default to 600 s CLONE_NEWNET inside _sandbox_preexec silently killed every outbound HTTP request from sandboxed Python whenever the kernel allowed unprivileged user namespaces. requests.get('https://huggingface.co'), urllib.request.urlopen('https://en.wikipedia.org/wiki/...'), socket.connect(('arxiv.org', 443)) all failed despite the AST visitor intending to allow them. The bash blocklist (curl / wget / nc / ssh / scp / sftp / rsync / socat / eval / source) plus the AST-level metadata-host denylist still carry the network policy after this change; CLONE_NEWNET was redundant with both. Add _TRUSTED_PUBLIC_HOST_LITERALS + _TRUSTED_PUBLIC_HOST_SUFFIXES (~100 informational hosts: Wikipedia language subdomains, Wikimedia, Wikidata, Google search, Bing, DuckDuckGo, HuggingFace, GitHub, raw.githubusercontent.com, arXiv, StackOverflow / Stack Exchange, MDN, docs.python.org, PyTorch / TensorFlow / NumPy / pandas docs, pypi / files.pythonhosted.org / npmjs / crates.io, ReadTheDocs, arXiv, Britannica, BBC / Reuters / Nature / Science, NASA / CDC / NIH / WHO open data, api.weather.gov). The visitor now blocks literal hosts that are neither metadata nor trusted with a short LLM-readable string so the model can retry with an allowed source instead of choking on a multi-line error. Block upload-shape calls regardless of host: requests.post / put / patch / delete / request with files= or data=open(...) / data=bytes_literal; httpx equivalents; urllib.request.urlopen / Request with data=...; HuggingFace upload_file / upload_folder / upload_large_folder / create_commit (module-level FQ paths AND method-name match on any receiver). Message: "Blocked: file upload disallowed in sandbox". Bump UNSLOTH_STUDIO_SANDBOX_CPU_S default 300 -> 600 s so long agentic chains that span multiple tool calls don't get SIGXCPU'd mid-stride. Env-var override path is unchanged. Host normalisation now strips trailing dot, userinfo @, and explicit port before allowlist / denylist comparison so trailing-DNS-dot, userinfo-smuggling, and explicit-:443 URLs are decided correctly. * studio: raise default request-body cap from 100 MB to 500 MB UNSLOTH_STUDIO_MAX_BODY_MB default goes 100 -> 500 to comfortably cover vision + audio + multi-recipe-batch JSON payloads. The MaxBodyMiddleware stream-counting logic from this branch's earlier 06ec088 already handles chunked bodies up to the new cap; env-var override path is unchanged for callers that want a tighter limit. * studio/auth: restore /api/auth/status.default_username to 'unsloth' This branch's earlier b39e9a4 changed default_username to None on the public /api/auth/status endpoint so the username field didn't leak to unauthenticated callers. In practice this regressed third-party clients (and the in-tree React login form's pre-fill UX) without adding meaningful security: the bootstrap password is the actual secret, and the username 'unsloth' is the documented default. Pin default_username to storage.DEFAULT_ADMIN_USERNAME ('unsloth') and tighten the response model so the field is required rather than Optional. Anyone who needs anonymisation can still reach for an allow-list deployment with auth disabled. * studio/training: raise max_seq_length / batch_size / lora_r / lora_alpha caps This branch's 7102815 introduced field validators with conservative caps. The follow-up loosens them so long-context experiments and high-rank LoRA exploration aren't gated at the schema layer: _MAX_BATCH_SIZE 1024 -> 4096 _MAX_SEQ_LENGTH 131_072 -> 2_000_000 (2M tokens) lora_r cap 512 -> 16_384 (_MAX_LORA_R) lora_alpha cap 1024 -> 32_768 (_MAX_LORA_ALPHA) _MAX_GRAD_ACCUM / _MAX_STEPS / _MAX_EPOCHS / lora_dropout / warmup_ratio / weight_decay are unchanged. Hardware (VRAM, host RAM, kernel launch latency) is now the binding constraint at the new caps, which is the correct ordering -- the validator stays a sanity check on -1 / 0 / 'abc' style garbage, not a usability gate. * studio/tests: cover sandbox allowlist + upload block + raised training caps studio/backend/tests/test_sandbox_tools.py (new): TestMetadataHostDenylist -- short "Blocked: cloud-metadata host" message on AWS IMDS, GCP metadata, Alibaba ECS, AWS IPv6 IMDS, 169.254/16. TestTrustedHostAllowlist -- Wikipedia (any language subdomain), Google, DuckDuckGo, HF, raw GitHub, arXiv, StackOverflow / family, MDN, docs.python.org, pypi, BBC, api.weather.gov, NumPy / PyTorch docs. TestUntrustedHostBlock -- example.com / random unlisted host rejected with the short "Blocked: host not in sandbox allowlist; use an allowed informational source" message. Dynamic URLs (computed var) still pass -- documented limit of static analysis. TestHostNormalization -- trailing dot, explicit :443, uppercase, userinfo-@-smuggle all decided correctly without false-block / false-pass. TestUploadDenylist -- requests / httpx / urllib.urlopen with files= / data=open / data=bytes, HfApi().upload_file / upload_folder / create_commit, module-level huggingface_hub.upload_folder. POST json= to trusted host still passes. TestSandboxCpuRlimitDefault -- pin UNSLOTH_STUDIO_SANDBOX_CPU_S=600 default and confirm CLONE_NEWNET source line is gone. TestMaxBodyDefault -- pin UNSLOTH_STUDIO_MAX_BODY_MB=500 default. studio/backend/tests/test_studio_train_validation.py (new): Pin at-cap-accepts / over-cap-rejects boundaries for max_seq_length=2_000_000, batch_size=4_096, lora_r=16_384, lora_alpha=32_768 so a future regression that tightens them back without explicit user opt-in is caught. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * studio: tighten code comments across the security-hardening pass * studio: always inject bootstrap credentials on first boot The UNSLOTH_STUDIO_INJECT_BOOTSTRAP gate added an extra terminal-to-browser copy-paste on every fresh install. In practice the LAN credential leak it guarded against is narrow: the password is one-time, the user rotates it on the very next click, the default Studio bind is 127.0.0.1, and -H 0.0.0.0 already exposes the entire API surface. Drop the gate so the inject fires whenever a bootstrap password is still pending. The CSP nonce wiring stays in place; the inline script remains the only inline script the backend ever emits. The three Playwright UI smoke workflows lose their UNSLOTH_STUDIO_INJECT_BOOTSTRAP=1 lines along with the explanatory comment blocks since the inject now happens by default. --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Wasim Yousef Said <wasimysdev@gmail.com>	2026-05-13 06:12:18 -07:00
Daniel Han	7be10852cb	install: support STUDIO_HOME / UNSLOTH_STUDIO_HOME for custom install paths (#5190) ... * install: support STUDIO_HOME / UNSLOTH_STUDIO_HOME for custom install paths Currently install.sh and install.ps1 hardcode all install paths off $HOME / $env:USERPROFILE with no env-var fallback. This blocks workspace-isolated installs (CI sandboxes, per-PR test environments, multi-tenant boxes) unless the entire HOME / USERPROFILE is faked, which also relocates ~/.gitconfig, ~/.ssh, and other unrelated state. Add an opt-in env-var override that does only what is needed. Resolution priority (highest first): 1. HOME / USERPROFILE explicitly redirected vs the password-database default. Detected via getent (Linux), dscl (macOS), or [Environment]::GetFolderPath (Windows). Best-effort: when the detection mechanism is unavailable the check is skipped and we fall through to step 2. 2. UNSLOTH_STUDIO_HOME, if set. 3. STUDIO_HOME, if set (alias for convenience; the variable name already matches the internal var install.sh sets). 4. Default: legacy $HOME/.unsloth/studio (or $USERPROFILE\.unsloth\studio on Windows). Identical to today's behavior when no env var is set. When an env var override fires: * DATA_DIR is nested inside ($STUDIO_HOME/share, or $StudioHome\share on Windows) so the runtime launcher and shortcuts find studio.conf in the same place install-time wrote it. * The unsloth CLI shim lands at $STUDIO_HOME/bin/unsloth (Unix) or $StudioHome\bin\unsloth.exe (Windows). On Windows the shim already lives under $StudioHome; the change only redirects DATA_DIR and skips the persistent registry PATH update. * Persistent shell PATH modifications are skipped (no .bashrc / .zshrc / .profile append on Unix; no Add-ToUserPath on Windows). Caller is expected to invoke via absolute path or add the bin dir to PATH explicitly. Avoids polluting the user's profile with a workspace-scoped path that may be deleted. The Unix launcher script is the only piece that must read DATA_DIR at runtime (it sources studio.conf from there). The hardcoded DATA_DIR inside the LAUNCHER_EOF heredoc is replaced with an @@DATA_DIR@@ placeholder substituted via sed at install time, using the same approach the script already uses for other install-time substitutions. Default path behavior is unchanged: when no env var is set and HOME is not redirected, install.sh / install.ps1 produce exactly the same file layout as today. Test scenarios verified locally on install.sh: * Default (no env vars) -> $HOME/.unsloth/studio (legacy) * HOME=/tmp/x -> /tmp/x/.unsloth/studio * UNSLOTH_STUDIO_HOME=/tmp/y -> /tmp/y as STUDIO_HOME root * STUDIO_HOME=/tmp/z (alias) -> /tmp/z as STUDIO_HOME root * HOME redirect + env var (HOME wins) -> install follows HOME * Unwritable override -> exits with clear ERROR message * install: priority change -- env vars now win over HOME redirect Flip the resolution order so explicit env vars take precedence over HOME / USERPROFILE redirection. New priority (highest first): 1. UNSLOTH_STUDIO_HOME, if set. 2. STUDIO_HOME, if set. 3. HOME / USERPROFILE explicitly redirected. 4. Default. Rationale: the env vars are explicit single-purpose signals (the user typed UNSLOTH_STUDIO_HOME=... specifically to redirect Studio). HOME redirection is broader and incidental -- the user may have redirected HOME for unrelated reasons (workspace tools, container builds) without wanting Studio to follow it. When both are set, the more specific signal should win. When only HOME is redirected (no env var), behavior is unchanged from the previous commit: install follows $HOME. * install: address review feedback (sed escape, downstream propagation, edge cases) Fixes from gemini-code-assist + chatgpt-codex-connector + reviewer.py 20-parallel run on the open PR. install.sh: * Escape sed replacement metacharacters before substituting @@DATA_DIR@@. Two-stage escape: ' -> '\'' for safe single-quote shell embedding, then \, &, | for sed replacement string + chosen delimiter. Heredoc switched to single-quoted DATA_DIR='@@DATA_DIR@@' so we only need single-quote escaping at runtime. Verified end-to-end with paths containing & and | (the sed delimiter). * Pass UNSLOTH_STUDIO_HOME into both setup.sh invocations (--local and PyPI paths) so the downstream install resolves the same Studio root install.sh picked. * macOS .app stub: replace hardcoded exec "$HOME/.local/share/unsloth/launch-studio.sh" with exec "$_css_data_dir/launch-studio.sh" so the .app launches the resolved launcher even in env-override mode. * Use mkdir -p -- and cd -- when validating the env override so paths starting with - cannot be misread as flags. install.ps1: * Drop .Guid from [guid]::NewGuid().Guid: the property does not exist; the probe filename was always identical and not unique. Default ToString() on System.Guid produces the canonical UUID string we want. * Guard LOCALAPPDATA before Join-Path to avoid aborting the installer in service / CI contexts where LOCALAPPDATA is unset (Join-Path under $ErrorActionPreference='Stop' would otherwise throw). Computed once into $defaultDataDir; both 'profile' and 'default' branches reuse it. * Set $env:UNSLOTH_STUDIO_HOME for the duration of the 'unsloth studio setup' subprocess so studio/setup.ps1 and unsloth_cli see the same install root install.ps1 picked. Restored in a finally block. studio/setup.sh: * Honor UNSLOTH_STUDIO_HOME / STUDIO_HOME (alias) when resolving STUDIO_HOME, VENV_DIR, VENV_T5_*_DIR. Falls back to the legacy $HOME/.unsloth/studio when no override is set. studio/setup.ps1: * Same change in PowerShell: honor $env:UNSLOTH_STUDIO_HOME / $env:STUDIO_HOME for $StudioHome / $VenvDir resolution. unsloth_cli/commands/studio.py: * Replace the module-level constant STUDIO_HOME = Path.home() / ".unsloth" / "studio" with a resolver that honors UNSLOTH_STUDIO_HOME / STUDIO_HOME before falling through to the legacy default. Same precedence the installers use. Verified locally: 6 install.sh scenarios still produce correct paths (default, HOME redirect, env var, alias, both, bad override). New sed-escape unit tests pass for paths containing & and |. Python resolver matches priority: UNSLOTH_STUDIO_HOME > STUDIO_HOME > default. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * install.sh: portable sed (no -i.bak) per gemini review feedback GNU sed -i.bak vs BSD/macOS sed -i.bak vs BusyBox sed have subtly different semantics. Use the POSIX-portable redirect-then-mv pattern instead. Functionally identical, runs everywhere. * studio: persist UNSLOTH_STUDIO_HOME so fresh shells find custom installs Without this, a custom-root install (UNSLOTH_STUDIO_HOME=/work/studio bash install.sh --local) only worked in the same shell that ran the installer. Closing the terminal and reopening lost the env var, the PATH was deliberately not persisted, and the Python CLI fell back to ~/.unsloth/studio. Result: 'Studio not set up' or quietly operating on a stale legacy install. Three persistence layers, all backwards-compatible (default installs emit zero changes): 1. Unix studio.conf install.sh now writes 'export UNSLOTH_STUDIO_HOME=...' next to UNSLOTH_EXE in studio.conf when in env-override mode. The launcher sources studio.conf at startup so the exec'd binary gets the var. Default installs do not write this line; studio.conf stays byte-identical to before. 2. Windows launch-studio.ps1 install.ps1 prepends '$env:UNSLOTH_STUDIO_HOME = ...' to the generated launcher when in env-override mode. Default installs produce the same launcher content as before. 3. Python sys.prefix inference storage_roots.studio_root() and unsloth_cli/commands/studio.py now infer the install root from sys.prefix when no env var is set (Path(sys.prefix).parent for unsloth_studio venvs). Catches direct invocations of <STUDIO_HOME>/bin/unsloth that bypass the launcher entirely. unsloth_cli/commands/studio.py also re-exports the resolved UNSLOTH_STUDIO_HOME via os.environ.setdefault so child processes (setup script, backend run.py) inherit it. Backend storage roots (storage_roots.studio_root, cache_root) now respect the env var via the shared resolver. run.py PID file, transformers_version.py T5 venvs, and model_config.py vision-check venv all switch to studio_root() so custom installs are self-contained. studio/setup.ps1: T5 sidecar venvs now resolve under $StudioHome (was $env:USERPROFILE\.unsloth\studio\.venv_t5_*). studio/setup.sh + studio/setup.ps1: llama.cpp build dir nests under $STUDIO_HOME / $StudioHome when env-override is active, otherwise keeps the legacy ~/.unsloth/llama.cpp. Verified locally: * studio.conf write block: env-override mode emits the export line; default mode does not (byte-identical to today). * PowerShell heredoc interpolation: correct output for both modes. * studio_root() resolver: default, UNSLOTH_STUDIO_HOME, STUDIO_HOME alias, and sys.prefix-based inference all return correct paths. * cache_root() now derives from studio_root(). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * install: tilde expansion + macOS .app stub safe-quoting Two fixes from running a 25-scenario simulation sweep against install.sh across path edge cases (spaces, apostrophes, ampersands, pipes, backslashes, dollar signs, Unicode, trailing slash, relative paths). 1. UNSLOTH_STUDIO_HOME=~/foo was landing as literal '~/foo' (env vars are not subject to tilde expansion). Added a POSIX-portable case block in install.sh, install.ps1, studio/setup.sh, studio/setup.ps1 that expands a leading ~ or ~/ to $HOME / $env:USERPROFILE. The prefix-removal pattern is single-quoted ('${var#'~/'}') so the shell does not tilde-expand the pattern back to $HOME/ before matching -- a subtle dash/bash gotcha. 2. macOS .app stub used an unquoted heredoc ('<< STUB_EOF'), so any $VAR / backtick / etc in the path would expand at .app launch time. Switched to single-quoted heredoc ('<< 'STUB_EOF'') with a placeholder + sed substitution + single-quoted shell embedding, matching the @@DATA_DIR@@ pattern already used for launch-studio.sh. Verified: 25/25 simulation scenarios pass on Linux dash + bash, including paths with $VAR, &, |, \\, ', spaces, and Unicode. End-to-end install in env-mode + fresh-shell launcher invocation confirmed: studio binds to /api/health from a clean env, and sys.prefix-based inference correctly returns the workspace root. * install: stop accidentally treating default installs as env-override Reviewer.py 20-runs cycle 1 found a unanimous P1 regression: a default 'unsloth studio update' relocates llama.cpp from ~/.unsloth/llama.cpp to ~/.unsloth/studio/llama.cpp, because the CLI was re-exporting UNSLOTH_STUDIO_HOME unconditionally and install.sh / install.ps1 were passing it into setup.{sh,ps1} unconditionally. The setup scripts treated the var's mere presence as "env-override mode" and relocated the llama.cpp build dir away from the legacy path, breaking the runtime backend's _find_llama_server_binary lookup on default installs. Fixes: * unsloth_cli/commands/studio.py: _resolve_studio_home now returns (path, is_custom). Re-export only when is_custom -- a real env override or a sys.prefix inference that resolves to a non-legacy path. Default installs leave UNSLOTH_STUDIO_HOME unset. * install.sh: gate UNSLOTH_STUDIO_HOME on $_STUDIO_HOME_REDIRECT == env before calling setup.sh. Use 'env $VARS bash setup.sh' so the var is set only for the subprocess, never leaked. * install.ps1: gate $env:UNSLOTH_STUDIO_HOME on $StudioRedirectMode -eq 'env' before invoking 'unsloth studio setup'. Restore prior value in finally block (unset if it wasn't set). * studio/setup.sh + setup.ps1: decide llama.cpp install root from the resolved $STUDIO_HOME (not from env-var presence). If the resolved path equals the legacy default ($HOME/.unsloth/studio), fall back to ~/.unsloth/llama.cpp. This makes setup robust against a stale UNSLOTH_STUDIO_HOME inherited from a parent process that happens to point at the legacy default. * studio/backend/core/inference/llama_cpp.py: - _find_llama_server_binary() now searches studio_root() / llama.cpp AND the legacy ~/.unsloth/llama.cpp (de-duped). Custom-root installs become discoverable; default installs unaffected. - kill_orphaned_servers ownership allowlist also includes studio_root() / llama.cpp so custom-root processes are cleanable. Verified locally: * 25/25 sim scenarios still pass (path edge cases unchanged). * setup.sh unit test: default-mode lands UNSLOTH_HOME at $HOME/.unsloth; env-mode lands at $STUDIO_HOME. * Python CLI unit test: default-mode returns is_custom=False and does NOT setdefault UNSLOTH_STUDIO_HOME; env-mode sets is_custom=True. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * install: || exit 1 on STUDIO_HOME subshell (dash set -e gap) Gemini review feedback: in dash, set -e does not trigger on subshell failures inside variable assignments. If 'cd -- "$_override" && pwd' fails, STUDIO_HOME stays empty and DATA_DIR collapses to /share. Add explicit '|| exit 1' on both install.sh:187 and setup.sh:413. * install.sh: argv-safe setup invocation for paths with spaces Cycle 2 reviewer.py 20-runs found a unanimous P1: passing the env-var through 'env $_STUDIO_ENV_FOR_SETUP' word-splits on whitespace, so a custom root like '/tmp/Unsloth Studio' becomes 'UNSLOTH_STUDIO_HOME= /tmp/Unsloth' followed by env trying to exec 'Studio'. Replaced with a tiny helper that prepends the env-var directly to the argv (no string-form intermediary), so spaces are preserved as a single argument. Default-mode invocation skips the env-var entirely. Verified: 'UNSLOTH_STUDIO_HOME=/tmp/test space/studio' now reaches setup.sh as a single value. * studio: tighten sys.prefix inference + Tauri env handling + llama.cpp env Cycle 3 reviewer.py findings (3 P1s converging): * sys.prefix inference too broad: a developer venv named 'unsloth_studio' was being treated as a custom Studio root. Narrow with an installer- sentinel check (presence of share/studio.conf or bin/unsloth shim inside the parent dir) in both unsloth_cli/commands/studio.py and studio/backend/utils/paths/storage_roots.py. * Tauri studio/src-tauri/src/process.rs::find_unsloth_binary() hardcoded ~/.unsloth/studio. Honor UNSLOTH_STUDIO_HOME / STUDIO_HOME (in that priority order) before falling back to legacy. * unsloth-zoo's GGUF export binds LLAMA_CPP_DEFAULT_DIR at import time from UNSLOTH_LLAMA_CPP_PATH. For env-override installs, persist UNSLOTH_LLAMA_CPP_PATH alongside UNSLOTH_STUDIO_HOME in studio.conf (Unix), in the generated PowerShell launcher (Windows), and via os.environ.setdefault in the Python CLI when running on a custom root, so GGUF export uses the custom-root llama.cpp build instead of the legacy ~/.unsloth/llama.cpp. Default behaviour unchanged: no env vars are written to studio.conf in default mode, no LLAMA_CPP_PATH is set, and the dev-venv inference falls through to legacy when no installer sentinels are present. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * studio: desktop_auth env-aware + legacy-root llama.cpp consistency - desktop_auth.rs: honor UNSLOTH_STUDIO_HOME / STUDIO_HOME for the .desktop_secret path so Tauri desktop login works against custom-root installs instead of always reading ~/.unsloth/studio/auth/. - install.sh / install.ps1 / unsloth_cli/commands/studio.py: when an env override resolves to the legacy default ($HOME/.unsloth/studio), set UNSLOTH_LLAMA_CPP_PATH to ~/.unsloth/llama.cpp (matching setup.sh / setup.ps1's legacy-equality branch). Previously the persisted value pointed at $STUDIO_HOME/llama.cpp, which was a non-existent location and broke unsloth-zoo's import-time GGUF binding for that edge case. * studio: tauri studio_root helper + marker-file persistence + ~ expansion Address cycle-5 reviewer findings: - Add studio/src-tauri/src/studio_root.rs: shared resolver with UNSLOTH_STUDIO_HOME / STUDIO_HOME (priority order), tilde expansion (~, ~/..., ~\...), installer-written marker fallback, then ~/.unsloth/studio. 5 unit tests cover the expansion paths. - Tauri lookups now go through the shared resolver: - process.rs::find_unsloth_binary - desktop_auth.rs::desktop_secret_path - main.rs::setup_logging (tauri.log under custom root) - commands.rs::open_logs_dir (opens custom root dir) - install.rs work_dir uses parent of resolved root (avoids creating a stray ~/.unsloth on a custom-root install) - install.sh / install.ps1 (env-mode only): write ~/.unsloth/studio-home marker so the desktop app launched from Finder/Start Menu (no shell env inheritance) still resolves the custom root. - install.sh / install.ps1 non-interactive completion: when StudioRedirectMode=env, print the absolute custom-root shim path since the persistent rc/registry PATH update is intentionally skipped in env-override mode. - unsloth_cli/commands/studio.py: replace setdefault() with truthy-check so a blank UNSLOTH_STUDIO_HOME / UNSLOTH_LLAMA_CPP_PATH in the parent env doesn't suppress the inferred custom root. 40/40 cargo test --bins pass. * studio: validate marker file + write in --tauri mode + propagate to subprocess Cycle-6 reviewer follow-ups: - studio_root.rs marker resolver now validates the persisted path before using it. A stale ~/.unsloth/studio-home pointing at a deleted/moved workspace is ignored (resolution falls back to the legacy default rather than hijacking it). Validation accepts share/studio.conf sentinel or bin/unsloth shim. Trailing newline strip uses trim_end_matches(['\n','\r']) so paths whose content legitimately has leading/trailing spaces survive. - install.sh / install.ps1: marker write moved out of the launcher generation path so it runs before the Tauri-mode early exit. Both shell-launcher and Tauri-installed env-mode roots now persist the marker. Removed the duplicate marker write that was previously inside install.ps1's $studioHomeExport block. - studio/src-tauri/src/install.rs: pass UNSLOTH_STUDIO_HOME to the installer subprocess (when not already in scope) so app-initiated repair / update flows reach the same root the running app uses. cargo test --bins -- --test-threads=1: 44/44 pass (4 new tests for marker validation: sentinel accepted, bin shim accepted, empty dir rejected, missing path rejected). * studio: fix Tauri legacy-fallback regression + stale marker cleanup Cycle-7 reviewer follow-ups (regression I introduced in cycle 6): - studio_root.rs: add StudioRootSource enum + resolve_studio_root_with_source(). Lets callers distinguish a real custom override (Env / Marker) from the legacy fallback (Default). - studio/src-tauri/src/install.rs: only forward UNSLOTH_STUDIO_HOME to the installer subprocess when the resolution source is Env or Marker. The Default fallback must NOT be passed -- install.sh / install.ps1 treat any non-empty UNSLOTH_STUDIO_HOME as env-override mode and would relocate DATA_DIR to $STUDIO_HOME/share and _LOCAL_BIN to $STUDIO_HOME/bin (regressing default Tauri repair / update flows from the legacy ~/.local/share/unsloth and ~/.local/bin). - install.sh / install.ps1: clear stale marker on default / HOME-redirect installs. A user who first installed with UNSLOTH_STUDIO_HOME=/work/studio then later reinstalls without env vars no longer has the desktop app hijacked by ~/.unsloth/studio-home pointing at the old custom root. - install.sh / install.ps1: when env mode wins over a redirected HOME / USERPROFILE, write the marker into the OS-reported real profile home (getent / dscl on Unix; [Environment]::GetFolderPath on Windows) so a later desktop launch from the user's normal session still finds it. Falls back to the current HOME / USERPROFILE. cargo test --bins -- --test-threads=1: 45/45 pass (1 new for the source enum invariants). * install: scrub stale marker from real-home on HOME-redirect cleanup Cycle-8 reviewer follow-up: the previous cleanup branch only removed \$HOME/.unsloth/studio-home, leaving a stale marker in the real password-database home after a prior env-mode install. A later default install with redirected HOME / USERPROFILE would still see the desktop app resolving the old custom root. - install.sh: compute the real password-database home (via getent / dscl) unconditionally, and scrub markers from BOTH \$HOME and the real-home in the default / HOME-redirect cleanup branch. - install.ps1: build a profile-candidate list (current USERPROFILE + OS-reported real profile) and remove markers from EVERY candidate in the default / profile-redirect cleanup branch. bash -n + cleanup smoke verified. * revert: drop Tauri env-var support + marker file mechanism Keep this PR scoped to shell installer + Python backend env-var support. Tauri desktop integration with custom Studio roots is deferred to a separate, focused PR. Reverts to pre-PR state: - studio/src-tauri/src/process.rs (find_unsloth_binary) - studio/src-tauri/src/desktop_auth.rs (auth_secret_path) - studio/src-tauri/src/main.rs (setup_logging tauri.log path) - studio/src-tauri/src/commands.rs (open_logs_dir) - studio/src-tauri/src/install.rs (work_dir + subprocess env) - studio/src-tauri/src/studio_root.rs DELETED Removes from install.sh / install.ps1: - ~/.unsloth/studio-home marker write/read/cleanup - HOME-redirect-aware marker location logic What this PR keeps (the original scope): - install.sh / install.ps1: UNSLOTH_STUDIO_HOME / STUDIO_HOME env-var resolver with HOME-redirect detection, tilde expansion, legacy fallback. Default installs are byte-identical to pre-PR. - studio/setup.sh / studio/setup.ps1: legacy-equality llama.cpp path. - studio.conf / launcher persists UNSLOTH_STUDIO_HOME + UNSLOTH_LLAMA_CPP_PATH for fresh shells (env-mode only). - unsloth_cli/commands/studio.py: env > sys.prefix sentinel > legacy resolver, conditional re-export. - studio/backend/utils/paths/storage_roots.py: same resolver. - Backend modules use storage_roots (run.py, model_config.py, transformers_version.py, llama_cpp.py). cargo test --bins -- --test-threads=1: 34/34 pass (pre-PR baseline). bash -n install.sh: clean. * install: cycle-10 fixes (default launcher, --tauri guard, env-mode shortcuts, win PATH) - install.sh launcher: default and HOME-redirect installs keep the legacy DATA_DIR=\"\$HOME/.local/share/unsloth\" runtime form so a later shell with a different \$HOME still resolves DATA_DIR. Only env-mode bakes the resolved absolute path. Restores byte-identical default behavior. - install.sh / install.ps1: fail fast when --tauri is combined with UNSLOTH_STUDIO_HOME / STUDIO_HOME. The desktop app still resolves the legacy ~/.unsloth/studio root, so a custom-root --tauri install would yield a desktop app that cannot find its binary or auth secret. Print the right alternative. - install.sh / install.ps1: skip persistent desktop / Start-Menu shortcuts in env-override mode. Workspace-scoped installs would otherwise leave launchers pointing at a path the user may delete. Default and HOME/profile-redirect installs keep the shortcut. - install.ps1: re-prepend env-override \$ShimDir AFTER Refresh-SessionPath. Refresh rebuilds PATH as Machine > User > current \$env:Path, so a previously-installed legacy User PATH entry would otherwise win precedence over the current-session env-override shim. bash -n install.sh, pwsh parser install.ps1 + setup.ps1: clean. cargo test --bins -- --test-threads=1: 34/34 (Tauri unchanged). * install: cycle-11 fixes (env-mode launcher writes, --tauri legacy passthrough, run.py llama path) - install.sh / install.ps1: env-mode no longer skips the entire create_studio_shortcuts / New-StudioShortcuts function. Move the early-return INSIDE those functions, just before the persistent desktop / Start-Menu shortcut creation. The runtime launcher (launch-studio.sh / launch-studio.ps1), studio.conf with UNSLOTH_STUDIO_HOME / UNSLOTH_LLAMA_CPP_PATH exports, and the icon ARE always written so env-mode shims can resolve via fresh shells. - install.sh / install.ps1: --tauri guard passes through when the override resolves to the legacy default ($HOME/.unsloth/studio / %USERPROFILE%\.unsloth\studio). The desktop app already uses that path, so explicit-equality is a supported edge case (matches the llama.cpp legacy-equality branch). - studio/backend/run.py: when launched directly (bypassing the unsloth CLI), set UNSLOTH_STUDIO_HOME and UNSLOTH_LLAMA_CPP_PATH before the rest of import chain runs so unsloth-zoo's import-time LLAMA_CPP_DEFAULT_DIR binding picks up the custom-root build. Only set when STUDIO_ROOT is a real custom override; legacy default installs leave them unset. bash -n install.sh, pwsh parser install.ps1: clean. python ast parse studio/backend/run.py: clean. cargo test --bins -- --test-threads=1: 34/34 pass (Tauri unchanged). * install: cycle-12 fixes (--tauri trailing slash + main.py uvicorn env) - install.sh / install.ps1 --tauri legacy passthrough: strip trailing separators before comparing the override to the legacy default. Previously UNSLOTH_STUDIO_HOME=\"\$HOME/.unsloth/studio/\" (with trailing slash) was rejected even though it resolves to the supported legacy root. - studio/backend/main.py: when launched directly via \`uvicorn main:app\` from a custom-root venv (bypassing both unsloth_cli and run.py), export UNSLOTH_STUDIO_HOME and UNSLOTH_LLAMA_CPP_PATH before any unsloth-zoo import so its import-time LLAMA_CPP_DEFAULT_DIR binding picks up the custom-root build. Only sets when STUDIO_ROOT is a real custom override. bash -n install.sh, pwsh parser install.ps1, python ast main.py: clean. Smoke probe: UNSLOTH_STUDIO_HOME=\$HOME/.unsloth/studio/ install.sh --tauri no longer exits with the unsupported-custom-root error. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * install.ps1: skip CWD-relative venv migration in env-override mode The legacy ~/unsloth_studio venv migration path on Windows reads %USERPROFILE%\unsloth_studio\Scripts\python.exe (a fixed home-relative path). Under env-override mode this would Move-Item the user's pre-existing default-install venv into $StudioHome\unsloth_studio, breaking the default install and contaminating the workspace root. Gate the migration on $StudioRedirectMode -ne 'env' so workspace-scoped installs leave the user's default-install venv untouched. No Linux equivalent: install.sh migrates from \$STUDIO_HOME/.venv which is already env-mode-aware (points at the workspace root, not \$HOME). * install: cycle-14 fixes (Tauri env scrub + setup.ps1 missing-root error) Tauri does not honor UNSLOTH_STUDIO_HOME / STUDIO_HOME / UNSLOTH_LLAMA_CPP_PATH yet -- the desktop app's Rust paths use the legacy ~/.unsloth/studio root. If the user's shell has these env vars set, spawned Python subprocesses would diverge from the Rust paths (custom-root Python <-> legacy-root Rust). Scrub the three env vars at all Tauri subprocess spawn sites: - process.rs: backend launch - desktop_auth.rs: provision-desktop-auth subprocess - install.rs: install.sh / install.ps1 invoked from the desktop app (also prevents the --tauri guard from rejecting an inherited override). setup.ps1: when UNSLOTH_STUDIO_HOME points at a non-existent directory, 'Resolve-Path -LiteralPath' threw a confusing PSObject error under $ErrorActionPreference = "Stop". Test-Path the override first and emit a friendly "run install.ps1 to create the install root" message instead. * install: cycle-15 fixes (preserve UNSLOTH_LLAMA_CPP_PATH + add update.rs scrub) UNSLOTH_LLAMA_CPP_PATH is a pre-existing custom-llama.cpp-directory override the Python backend (studio/backend/core/inference/llama_cpp.py) and unsloth-zoo intentionally support. It is unrelated to the Studio install root. Cycle 14 over-scrubbed it from the Tauri spawn sites, regressing desktop GGUF/llama.cpp workflows for users who set it in their shell. - process.rs / desktop_auth.rs / install.rs: stop scrubbing UNSLOTH_LLAMA_CPP_PATH; only scrub UNSLOTH_STUDIO_HOME and STUDIO_HOME. - update.rs: missed Tauri spawn site -- add the same UNSLOTH_STUDIO_HOME / STUDIO_HOME scrub so 'unsloth studio update' from the desktop app updates the legacy-root install Tauri actually manages. Verified: cargo test --bins -- --test-threads=1 -> 34/34 pass. * install.sh: document apostrophe-escape derivation inline The shell quoting at install.sh:642 / 659 / 679 / 680 / 823 has been flagged as broken across multiple review cycles, but every end-to-end verification (DATA_DIR=\"a b's&c|d\$e\" -> generated launcher -> source -> recovered exact input) passes. The proposed "8 backslash" fix would double the escape and actually break what currently works. Strengthen the inline comments to spell out the derivation: - shell pattern \"s/'/'\\\\''/g\" passes \"s/'/'\\''/g\" to sed (\\\\ -> \\) - sed replacement '\\'' yields close-quote / escaped-quote / open-quote - stage 2 (\\, &, |) only needed where the value is then sed-replaced into a launcher template via s|@@DATA_DIR@@|VALUE|g studio.conf is written via printf, not sed, so it only needs stage 1. No behavior change, only inline doc to head off future false positives. * install/setup .ps1: use -LiteralPath for $StudioHome-derived paths Pre-PR, $StudioHome was hardcoded to %USERPROFILE%\.unsloth\studio -- no wildcard characters possible. The PR introduces UNSLOTH_STUDIO_HOME / STUDIO_HOME, so $StudioHome (and every path derived from it: $VenvDir, $VenvPyExe, $UnslothExe, $UnslothHome, $LlamaCppDir, $VenvT5_*, etc.) can now contain bracket characters that PowerShell would interpret as wildcards. Reproducer (from cycle 17 review 20): pwsh> Test-Path 'studio[abc]/Scripts/python.exe' False pwsh> Test-Path -LiteralPath 'studio[abc]/Scripts/python.exe' True Switch the relevant Test-Path / Remove-Item / New-Item / Move-Item calls in install.ps1 and studio/setup.ps1 to -LiteralPath. Sites where the path is fixed (the shim under %LOCALAPPDATA%\Microsoft\WindowsApps, $RepoRoot from -PSCommandPath) keep the wildcard-aware form. * install/setup .ps1: fix New-Item -LiteralPath regression from cycle 17 Cycle 17 added -LiteralPath to all $StudioHome-derived path operations, but New-Item has no -LiteralPath parameter (verified pwsh 7.6 syntax: "New-Item [-Path] <string[]> [-ItemType <string>] ..."). Every directory- creation site would throw "A parameter cannot be found that matches parameter name 'LiteralPath'" at runtime, blocking T5 sidecar setup, llama.cpp parent creation, and StudioHome creation. Likewise, "Split-Path -LiteralPath $X -Parent" cannot mix LiteralPath with -Parent (separate parameter sets). The default LiteralPath mode already returns the parent. Switch to [System.IO.Directory]::CreateDirectory($X), which natively takes a literal path, and drop the trailing -Parent on Split-Path. Verified end-to-end on a bracketed path "/tmp/...[abc]": - CreateDirectory: created - Test-Path -LiteralPath: detects - nested CreateDirectory(Split-Path -LiteralPath ...): works * install/setup .ps1: extend -LiteralPath sweep to remaining \$StudioHome paths Cycle 17/18 missed several wildcard-aware operations on user-controlled \$StudioHome-derived paths. Reviewers identified remaining sites: install.ps1: - \$UnslothExePath (Test-Path / Resolve-Path) at the shortcut creator - \$VenvDir (Get-ChildItem) at the no-torch-runtime resolver - \$ShimDir (New-Item Directory -- replaced with .NET CreateDirectory) - \$ShimExe (Test-Path / Remove-Item / re-prepend guards) -- the shim lives at \$StudioHome\\bin\\unsloth.exe in env-override mode, so it inherits bracket sensitivity from \$StudioHome. - \$UnslothExe (Copy-Item fallback) when HardLink fails. studio/setup.ps1: - \$LlamaServerBin (Test-Path) at the prebuilt-bundle / source-build validation gates (3 sites). \$LlamaServerBin lives under \$BuildDir under \$LlamaCppDir under \$UnslothHome under \$StudioHome. New-Item HardLink keeps -Path because creating a non-existent target with brackets succeeds (verified via direct pwsh smoke test). * install: cycle-20 fixes (more setup.ps1 -LiteralPath + shell-quote launch hints) setup.ps1: extend -LiteralPath sweep to remaining \$BuildDir-derived paths that the cycle-19 commit missed: - \$CmakeCacheFile (Test-Path + Select-String -Path) - \$buildTmp (10 Test-Path / Remove-Item sites in source-build cleanup) - \$QuantizeBin (Test-Path) - \$altBin (Test-Path) These all live under \$BuildDir -> \$LlamaCppDir -> \$UnslothHome -> \$StudioHome, which is now user-controlled via UNSLOTH_STUDIO_HOME. Bracket characters in the override would silently skip rebuild detection or leave stale build artifacts. install.sh: shell-quote the launch-instruction substep lines for env- override mode. UNSLOTH_STUDIO_HOME values containing spaces or apostrophes (e.g. "/tmp/O'Brien Studio") would print copy-paste- unsafe commands -- the install succeeded but the printed launch instructions split at the space. Now wraps with the canonical '\\''-style escape so the printed lines parse with bash -n. Verified end-to-end: - printed shim line: '/tmp/O'\''Brien Studio/bin/unsloth' studio ... - bash -n on the printed line passes. * install.ps1: -LiteralPath for macOS-stub-launcher \$appDir-derived paths The shortcut/launcher generator at install.ps1:418-693 writes the stub launcher, .vbs, and icon under \$appDir = \$StudioDataDir, which in env-override mode is \$StudioHome\share. Cycle 17/19/20 missed the following wildcard-aware ops on these paths: - Test-Path \$appDir (with New-Item Directory swap to .NET CreateDirectory) - Set-Content -Path \$launcherVbs (for the WSH .vbs stub) - Test-Path / Copy-Item \$bundledIcon (bundled icon copy) - Test-Path / Remove-Item \$iconPath (icon header validation) In env-override mode \$StudioHome can contain bracket characters; without -LiteralPath the .vbs write fails outright and the icon validation can either skip a present icon or fail to delete a malformed one. (The COM shortcut creation downstream returns early in env-override mode, so its path values don't need this treatment.) * install: don't override pre-existing UNSLOTH_LLAMA_CPP_PATH in launchers Cycle 14/15 established UNSLOTH_LLAMA_CPP_PATH as a pre-existing custom-llama.cpp-directory override the Python backend and unsloth-zoo intentionally support, independent of the Studio install root. The launchers (studio.conf sourced by Unix launch-studio.sh, and the PowerShell launch-studio.ps1) were unconditionally re-exporting it, which silently overrides a user's pre-existing value when they invoke the launcher from a shell where UNSLOTH_LLAMA_CPP_PATH is already set. Make the assignment conditional in both launchers: install.sh studio.conf: if [ -z "\${UNSLOTH_LLAMA_CPP_PATH:-}" ]; then export UNSLOTH_LLAMA_CPP_PATH='...' fi install.ps1 launch-studio.ps1: if (-not \$env:UNSLOTH_LLAMA_CPP_PATH) { \$env:UNSLOTH_LLAMA_CPP_PATH = '...' } UNSLOTH_STUDIO_HOME stays unconditional: the launcher is bound to a specific install, so its STUDIO_HOME must always match that install. * install.sh: harden --tauri legacy resolver against CDPATH and symlinks Reviewer cycle 23 (inst 19) noted that the bare \`cd -- ... && pwd\` form in the --tauri legacy comparison can echo a CDPATH-prefixed path when the user has CDPATH set in their environment, contaminating the resolved absolute path used in the legacy-equality check. Switch to \`CDPATH= cd -P -- ... && pwd -P\` so: - CDPATH= clears the cd-prefix-echo behavior - -P / pwd -P resolves any symlinks to a canonical path No behavior change for users without CDPATH set; correctness fix for users who have it set in their shell. * install + llama_cpp backend: cycle-24 hardening Three real findings from cycle 24 reviewers: 1. install.sh:231 + studio/setup.sh:413 -- main \$STUDIO_HOME resolvers used the same bare \`cd -- ... && pwd\` form that cycle 23 only fixed for the --tauri guard. Switch both to: \$(CDPATH= cd -P -- "\$override" && pwd -P) so relative custom-root values don't get CDPATH-prefixed or have the cd-on-CDPATH stdout newline contaminate the captured value. 2. install.sh --tauri legacy root used logical \$HOME/.unsloth/studio while the override side was canonicalized via pwd -P. A symlinked \$HOME (e.g. /home/alice -> /u/alice) made the comparison fail even when both sides pointed at the same directory. Canonicalize the legacy side too when the dir exists. 3. studio/backend/core/inference/llama_cpp.py:_find_llama_server_binary searched \$STUDIO_HOME/llama.cpp first then ~/.unsloth/llama.cpp in default-mode installs. setup.sh / setup.ps1 only install llama.cpp under \$STUDIO_HOME/llama.cpp in env-override mode; in default mode it always lives at ~/.unsloth/llama.cpp. The post-PR search would pick up a stale partial install at ~/.unsloth/studio/llama.cpp over the real legacy binary. Mirror setup's legacy-equality check: when studio_root() resolves equal to ~/.unsloth/studio, search ONLY the legacy ~/.unsloth/llama.cpp. Otherwise (env-override custom root), search custom first, legacy fallback. * install + setup: canonicalize legacy-equality comparison sites Cycle 24 made \$STUDIO_HOME canonical via 'CDPATH= cd -P -- ... && pwd -P', but the legacy-equality comparison sites still used the bare logical "\$HOME/.unsloth/studio" string. With a symlinked \$HOME (e.g. /home/alice -> /u/alice), the comparison fails even when both sides point at the same dir, and llama.cpp ends up under a custom-root path the Python backend's legacy comparison cannot find. Reviewer cycle 25 inst 2 reproduced this with HOME=/tmp/link -> /tmp/real and UNSLOTH_STUDIO_HOME=\$HOME/.unsloth/studio: setup.sh resolves UNSLOTH_HOME to /tmp/real/.unsloth/studio while the backend search resolves both physically equal and looks at /tmp/link/.unsloth/llama.cpp. Canonicalize the legacy side at all four sites: - install.sh:695 (create_studio_shortcuts llama.cpp path) - studio/setup.sh:577 (UNSLOTH_HOME selection) - install.ps1:462 (launcher UNSLOTH_LLAMA_CPP_PATH path) - studio/setup.ps1:1829 (UnslothHome selection) Apply CDPATH= cd -P -- ... && pwd -P (Unix) or Resolve-Path -LiteralPath (Windows) when the legacy dir exists. unsloth_cli/commands/studio.py already does this via Path.resolve(). * llama_cpp: gate _kill_orphaned_servers studio-root allowlist on env-override Cycle 24 fixed _find_llama_server_binary to only search \$STUDIO_HOME/llama.cpp when STUDIO_HOME is a real env override (not the legacy default), but the symmetric _kill_orphaned_servers allowlist still appended _sr() / "llama.cpp" unconditionally. In default mode _sr() resolves to ~/.unsloth/studio, so ~/.unsloth/studio/llama.cpp would be treated as a Studio-owned install root for the orphan-kill scan even though the default installer does not own that path. A llama-server process running there from a different tool or a stale partial install would be killed. Apply the same legacy-equality check used in _find_llama_server_binary and the install/setup scripts: only add _sr()/"llama.cpp" to the allowlist when STUDIO_HOME != legacy default. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * setup.sh + setup.ps1: canonicalize both sides of legacy-equality check Proactive audit pass found one real asymmetry the cycle-by-cycle review process had not yet flagged: - install.sh:704 / install.ps1:469 are gated on env-mode and only run when STUDIO_HOME has already been canonicalized (cycle 24). Symmetric. - studio/setup.sh:577 / studio/setup.ps1:1829 run UNCONDITIONALLY, including in default mode. In default mode STUDIO_HOME is set to the bare logical \$HOME/.unsloth/studio (setup.sh:416) or Join-Path \$env:USERPROFILE ".unsloth\\studio" (setup.ps1:1480). Cycle 25 canonicalized only the legacy side, creating an asymmetry under symlinked \$HOME / junctioned %USERPROFILE%. Result of the asymmetry: a default-mode install on a host with \$HOME=/tmp/link -> /tmp/real treats the legacy default as a custom root, putting llama.cpp at \$STUDIO_HOME/llama.cpp instead of ~/.unsloth/llama.cpp -- and the Python backend's _find_llama_server_binary (which uses .resolve() on both sides) then can't find the install. Fix: canonicalize STUDIO_HOME on the fly at the comparison site, in both setup.sh and setup.ps1. Symmetric with the now-canonicalized legacy side from cycle 25, regardless of which mode set STUDIO_HOME. The other two comparison sites (install.sh:704, install.ps1:469) are already symmetric because they only run when STUDIO_HOME comes from the env-override resolution path that already does pwd -P / Resolve-Path. unsloth_cli/commands/studio.py + studio/backend/run.py + main.py + llama_cpp.py already use .resolve() on both sides -- symmetric. * install.ps1: env-override resolution uses .NET API for literal paths Gemini code-review (review 4177641398, commit 2ea2c91) caught two remaining New-Item -Path sites in the env-override resolution block that the cycle 18 sweep missed: - Line 123: New-Item -ItemType Directory -Path \$envOverride - Line 132: New-Item -ItemType File -Path \$probe (writability test) Both use -Path which interprets square brackets as wildcards. For a user with UNSLOTH_STUDIO_HOME=C:\\workspaces\\studio[abc], both calls would fail before the install starts. New-Item also has no -LiteralPath in PowerShell 5.1. Replace both with the .NET API: - [System.IO.Directory]::CreateDirectory(\$envOverride) - [System.IO.File]::WriteAllText(\$probe, "") -- closes the file handle before the Remove-Item below. End-to-end verified with /tmp/test-envoverride-[abc]-* path: CreateDirectory + WriteAllText + Test-Path -LiteralPath all work. * comments: condense multiline blocks added by this PR Across the 27-cycle review process, comments accumulated as multiline blocks explaining each fix's history (cycle numbers, prior bugs, reviewer rationale). Compress every block to 1-2 lines that capture just the WHY, dropping cycle references and history that belongs in the PR description / commit log instead. Net: 268 deletions / 124 insertions (-144 lines) of comments only. Behavior unchanged. Verified: bash -n, pwsh parser, python ast.parse, cargo check all pass. * install.ps1: use 'return' over 'exit 1' for Install-UnslothStudio bail-outs Per Gemini review #4177659001: when users run install.ps1 via 'irm ... | iex', 'exit 1' inside the function terminates the entire PowerShell process and closes the user's terminal. 'return' bails out of the function while keeping the shell open, matching existing error sites at lines 34, 50, 57. Three sites fixed: --tauri+env-override guard, env-override mkdir/access failure, and write-probe failure. The 'exit' calls at lines 591/611 are inside a generated launcher here-string (a separate top-level .ps1 that runs as its own process), so they correctly stay as 'exit'. * install.{sh,ps1}: address Gemini review #4177680451 Three medium fixes: 1. install.sh redirection detection: canonicalize both sides of the $HOME vs passwd-DB comparison via 'CDPATH= cd -P -- ... && pwd -P' so a trailing slash on $HOME (or symlink-vs-realpath mismatch with getent/dscl output) doesn't misfire the redirection branch. 2. install.sh shim symlink: 'ln -sf' into an existing directory creates the link INSIDE it ($_LOCAL_BIN/unsloth/unsloth instead of the intended file). Pre-strip a real (non-symlink) directory at $_LOCAL_BIN/unsloth before linking. 3. install.ps1 ShimExe: add -Recurse to Remove-Item so the launcher refresh recovers if $ShimExe somehow exists as a directory rather than a file (would otherwise drop into the catch and skip the shim update). * install.ps1: use 'throw' over 'return' for fatal validation failures Cycle 28 reviewer.py (12/8 RC/APPROVE) caught a regression introduced by the previous Gemini-review fix (#4177659001 -> commit 393e676b). 'return' inside Install-UnslothStudio kept iex'd terminals alive but made 'pwsh -File install.ps1' exit with code 0 on fatal validation failures (--tauri+custom-root rejected, STUDIO_HOME unwritable, etc.), so CI / wrapper scripts treated failed installs as successful. 'throw' satisfies both constraints: - pwsh -File install.ps1: exits with code 1 (CI sees failure) - irm | iex: shows error to user, does NOT close the host terminal Three sites: --tauri+env-override guard, mkdir/access failure, write-probe failure. Verified throw -> exit code 1 under pwsh -File. * install.ps1 launcher: single-quote child -Command path Cycle 28 P2 finding: the generated launch-studio.ps1 builds the child PowerShell -Command string with the executable path inside double quotes, so a custom Studio root containing PowerShell metacharacters (\$, backtick) re-expands in the child shell. Example: D:\work\\\$job\studio -> child reparses \$job and runs the wrong path. Fix: single-quote the path inside the child command and double any apostrophes (PowerShell's literal-quote-escape form) so paths like "O'Brien Studio & x|y" or "C:\work\\\$bad\studio" survive verbatim. * install: harden custom Studio root handling - install.sh shim refresh: refuse to recursively delete a real directory at $_LOCAL_BIN/unsloth before creating the symlink. The previous rm -rf could destroy unrelated user data living at that path. - install.ps1 shim refresh: drop -Recurse from Remove-Item on $ShimExe and refuse early when the shim path is a directory; mirrors the install.sh guard so a directory at $StudioHome\bin\unsloth.exe is not blown away. - install.ps1 PATH wiring: remove the redundant first $ShimDir prepend in env-override mode; the post-Refresh-SessionPath prepend is the one that takes effect, and the duplicate left $ShimDir in $env:Path twice. - install.ps1 manual launch instructions: single-quote the printed shim and Activate.ps1 paths so '$' / backtick metacharacters in custom roots do not reparse when the user copies and pastes the command. - studio/setup.sh: validate writability of UNSLOTH_STUDIO_HOME with the same [ -w ] check install.sh already has, so a read-only override fails with a clear message instead of an obscure uv pip permission error. - Drop the STUDIO_HOME alias everywhere (storage_roots.py, studio.py, install.sh, studio/setup.sh, install.ps1, studio/setup.ps1). The name is too generic and an ambient STUDIO_HOME from unrelated tooling could silently redirect the install. Only UNSLOTH_STUDIO_HOME is honored. - unsloth_cli/commands/studio.py: defer UNSLOTH_STUDIO_HOME / UNSLOTH_LLAMA_CPP_PATH re-export from import time into a helper invoked by the studio app callback. Importing the module no longer mutates os.environ as a side effect, so test runners and CLI introspection stop leaking those vars into unrelated subprocesses. - studio/backend/core/inference/llama_cpp.py: replace set-mutation inside list comprehension with an explicit dedup loop for readability. * install: harden custom Studio root edge cases - install.ps1 shim refresh: move the directory-collision preflight outside the lock-handling try/catch. The previous throw inside the try block was swallowed by the surrounding catch and downgraded to a "Continuing with the existing launcher" warning, leaving the install in a broken state with no usable shim on disk. - storage_roots.py / unsloth_cli/commands/studio.py: tighten the bin-shim sentinel from .exists() to .is_file(). A directory at the candidate bin/unsloth (or bin/unsloth.exe) path would otherwise false-positive the venv inference and pick the wrong Studio root. - storage_roots.py / unsloth_cli/commands/studio.py: wrap the env-var override Path(...).expanduser().resolve() in try/except (OSError, ValueError), matching the defensive pattern already used in studio/backend/main.py and studio/backend/run.py. An invalid override (unresolvable network drive, bad characters) now falls back to the un-resolved path instead of crashing at import time. * install: fail fast on missing custom root, allow brackets in shim path - install.ps1 shim hardlink: switch the New-Item -ItemType HardLink call from -Path to -LiteralPath so a custom Studio root containing bracket characters does not fail under PowerShell's wildcard-aware -Path parameter. Matches the -LiteralPath usage on every other Test-Path / Remove-Item / Copy-Item call against the same shim path. - studio/setup.sh override branch: replace the silent mkdir -p of the override directory with an existence check that exits 1 with a clear message. setup.sh runs against an existing install (via 'unsloth studio update'), so a typo in UNSLOTH_STUDIO_HOME must not materialize an empty workspace dir. Brings the Unix flow in line with setup.ps1, which already errors on a missing override root. * llama_cpp: scope orphan-server kill to the active install root _kill_orphaned_servers used to unconditionally include the legacy ~/.unsloth/llama.cpp tree in install_roots, even when the running Studio is in env-override mode and operates out of a custom root. On a single OS user running both a default-install Studio and a custom-root Studio concurrently, the custom Studio would kill the default Studio's llama-server during startup orphan cleanup. Hoist _is_custom_root out of the import try/catch so the legacy- append decision sees it (default to False on ImportError so default mode behaviour is unchanged), and gate the legacy ~/.unsloth/llama.cpp append on `not _is_custom_root`. * install: harden custom-root .venv migration and shim hardlink - install.sh / install.ps1 OLD-layout .venv migration: gate on default-mode only. Without the guard, pointing UNSLOTH_STUDIO_HOME at a workspace that already has .venv (e.g. an unrelated Python project) caused the torch validation to fail and the installer to recursively remove the user's project venv. Mirrors the existing env-mode skip on the CWD-relative venv migration immediately below. - install.ps1 shim hardlink: revert to New-Item -ItemType HardLink -Path. -LiteralPath is not accepted on the HardLink ItemType in any PowerShell version, so the previous form always threw and silently fell back to Copy-Item, breaking hardlink-update propagation. Bracket characters in $ShimExe are still defended by the directory-collision preflight added earlier. - storage_roots.py / unsloth_cli/commands/studio.py: strip whitespace from the UNSLOTH_STUDIO_HOME env var before the truthy check so a blank " " override does not become a real path with trailing spaces (which would silently break every downstream Studio path operation). * Studio paths: tolerate stat / resolve failures during root inference - storage_roots._infer_studio_home_from_venv: wrap the share/studio.conf and bin/shim is_file() sentinel checks in try/except OSError. A PermissionError on a restricted candidate dir would otherwise propagate out of studio_root() and crash module import in run.py / main.py / transformers_version.py / model_config.py at server startup. - llama_cpp._kill_orphaned_servers: broaden the studio_root() guard from ImportError-only to (ImportError, OSError, ValueError) so transient resolve / sentinel failures do not crash the orphan-killer at server startup. Matches _find_llama_server_binary's existing pattern. - llama_cpp._find_llama_server_binary: nest the inner resolve() in its own try/except and fall back to unresolved-path comparison instead of dropping the custom search root entirely. A transient resolve() error on the legacy path no longer loses the custom-root llama.cpp lookup. * Add Studio install-root resilience tests * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: isolate custom-root installs from default-install state - llama.cpp discovery in env-override mode no longer falls back to the legacy ~/.unsloth/llama.cpp tree. The orphan-cleanup path already excludes that root in custom mode; aligning discovery prevents a custom-root Studio from launching a sibling install's binary it then refuses to manage. Users who want a shared build set UNSLOTH_LLAMA_CPP_PATH explicitly. - Generated POSIX launcher (install.sh heredoc) namespaces LOCK_DIR with a hash of DATA_DIR and persists the launched port to $DATA_DIR/studio.port; in env-override mode the fast-path attaches only to a port we ourselves wrote, never to a sibling Studio that happens to be healthy on 8888..8908. - Generated Windows launcher (install.ps1 heredoc) bakes a per-install $portFile and SHA-256-suffixed mutex name, mirroring the POSIX side; Find-HealthyStudioPort uses the port file in env-override mode. - studio/setup.sh and studio/setup.ps1 require an .unsloth-studio-owned marker before deleting $STUDIO_HOME/.venv_t5*, $STUDIO_HOME/llama.cpp, and the sidecar T5 venvs in env-override mode. The marker is dropped after fresh creation so subsequent runs of 'unsloth studio update' proceed cleanly. Mirrors the existing .venv guard in install.sh. - Wrap bare Path.resolve() calls on the legacy STUDIO_HOME constant in studio/backend/main.py, studio/backend/run.py, and unsloth_cli/commands/studio.py in the same try/except (OSError, ValueError) used adjacently, so a restricted parent or recursive symlink on $HOME does not crash module import / CLI startup. * Studio: guard env-mode workspace against destructive cleanup - install.sh and install.ps1 unconditionally rm -rf / Remove-Item the new-layout $STUDIO_HOME/unsloth_studio when it has a python; in env-override mode that path is a user-chosen workspace, mirroring the .venv migration concern the .venv branch already guards. Refuse to remove an existing $STUDIO_HOME/unsloth_studio that lacks Studio sentinels (share/studio.conf or bin/unsloth). - studio/setup.ps1 only checked Test-Path -PathType Container on the custom root; setup.sh and install.ps1 both also write-probe via WriteAllText / Remove-Item. Add the matching probe so 'unsloth studio update' against an ACL-restricted root fails fast with a clear message instead of erroring later while creating sidecar venvs. * Add Studio install/setup workspace-isolation tests * Studio: tighten installer rationale comments - install.sh: collapse a 5-line restatement into 3 lines, naming env-mode behavior up front and the byte-identical pre-override fallback after. - install.ps1: correct misleading hardlink comment that claimed the directory-collision preflight guards against wildcard expansion; bracket characters in $ShimExe still glob-expand here, with the Copy-Item -LiteralPath fallback handling them. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Split: keep only 2 file(s) * Studio: harden env-mode workspace guards across installers and update path Tightens the UNSLOTH_STUDIO_HOME custom-root protections so destructive installer paths cannot displace unrelated user data when the override points at a workspace. install.sh / install.ps1: env-mode sentinel that gates rm -rf $VENV_DIR / Remove-Item $VenvDir now requires share/studio.conf or the bin/unsloth(.exe) shim to be a real file or symlink. Previously a directory at bin/unsloth or bin\unsloth.exe satisfied the check (-e and bare Test-Path accept any path type), so a workspace with unrelated content under unsloth_studio plus a sibling directory at bin/unsloth could be wiped. studio/setup.ps1: stale-venv rebuild branch now mirrors install.ps1's env-mode guard before Remove-Item -LiteralPath $VenvDir -Recurse -Force. Without this, "unsloth studio update" pointed at a custom workspace whose unsloth_studio venv fails torch validation deletes the venv even when the root carries no Studio sentinels. studio/setup.sh / studio/setup.ps1: prebuilt llama.cpp install path now calls _assert_studio_owned_or_absent / Assert-StudioOwnedOrAbsent before invoking install_llama_prebuilt.py, and writes the .unsloth-studio-owned marker on success. install_llama_prebuilt.py uses os.replace() to move any existing install_dir aside before staging, so an unrelated $STUDIO_HOME/llama.cpp could otherwise be displaced before the existing source-build ownership guard ever ran. * Studio: gate ownership guards on canonical custom-root and add venv marker Tightens UNSLOTH_STUDIO_HOME ownership semantics so they fire only for a genuinely custom root, never for an explicit override that resolves to the legacy default. Adds an in-VENV marker that lets a partial install be repaired and provides a strong primary sentinel for the deletion guard. studio/setup.sh + studio/setup.ps1: hoist the canonical $STUDIO_HOME vs legacy-default comparison so it sits next to the marker definition, derive _STUDIO_HOME_IS_CUSTOM / $StudioHomeIsCustom once, and gate the _assert_studio_owned_or_absent / Assert-StudioOwnedOrAbsent helpers and the prebuilt llama.cpp marker writes on that flag instead of raw env-var presence. UNSLOTH_STUDIO_HOME=$HOME/.unsloth/studio (legacy override) no longer trips the guard for pre-PR T5 sidecar venvs or llama.cpp dirs that predate the .unsloth-studio-owned marker. The duplicate canonical block inside the llama.cpp section is removed; the new flag is reused. studio/setup.ps1: Assert-StudioOwnedOrAbsent's marker check now requires -PathType Leaf so a directory at .unsloth-studio-owned cannot satisfy it. The in-place git-sync branch in the source-build path now calls Mark-StudioOwned after a successful sync so a later prebuilt-update path does not fail Assert-StudioOwnedOrAbsent on the same root. install.sh + install.ps1: write $VENV_DIR/.unsloth-studio-owned right after uv venv succeeds and accept it as the primary sentinel in the env-mode deletion guard. This recovers from a partial install that was previously unrepairable, and is a stronger sentinel than sibling shim files (the marker is inside the venv that is about to be wiped, so an unrelated workspace cannot accidentally satisfy it). install.sh: drop the standalone -L test on $STUDIO_HOME/bin/unsloth in the deletion guard. -L returns true for any symlink including symlinks to directories and broken symlinks; -f already accepts the legitimate file-targeted symlink shape created by ln -s at install.sh:1864. * Studio: close residual workspace-isolation gaps for custom roots Four follow-on hardenings that close the remaining cross-root leaks the custom-root install plumbing still left open. studio/setup.ps1 in-place git-sync: when the source-build path finds an existing $LlamaCppDir/.git, it ran git remote set-url, checkout -B, and clean -fdx in place before any ownership check. The previous fix marked the tree as Studio-owned AFTER the sync but did not guard the BEFORE case, so an unrelated workspace .git could be silently rewritten on the first source-build under a custom UNSLOTH_STUDIO_HOME. Add the same Assert-StudioOwnedOrAbsent guard already used by the prebuilt path and the temp-dir swap path (gated on $StudioHomeIsCustom for parity). Launcher port-file workspace isolation: the env-mode launchers' fast path attached to any backend listening on the cached port that returned a healthy /api/health, even when that backend belonged to a different install root. studio/backend/main.py /api/health now returns the resolved studio_root; install.sh _check_health and install.ps1 Test-StudioHealth verify it against UNSLOTH_STUDIO_HOME when set, so a stale studio.port pointing at a sibling Studio is rejected instead of opening the wrong UI. studio/src-tauri preflight + commands: the Tauri desktop app stays on the legacy root by design. process.rs / install.rs / desktop_auth.rs / update.rs already strip UNSLOTH_STUDIO_HOME and STUDIO_HOME from their CLI subprocesses, but preflight.rs run_cli_probe / probe_cli_capability and commands.rs check_install_status did not, so a desktop launch from a shell carrying those env vars produced status reflecting a different root than the desktop manages. Mirror the existing scrub. install.sh shim install: the previous `rm -f -- $_shim_path; ln -s ...` pair leaves a window with no shim if interrupted. Use ln -sfn for an atomic replace; the -n flag prevents descent into a symlink-to-directory target (the existing directory guard above already rejects a real dir). * Studio: replace launcher root verify with hex digest baked at install time The previous launcher identity check returned the absolute resolved Studio install root from /api/health and matched it against $UNSLOTH_STUDIO_HOME in the launcher. Three problems that this commit closes: - POSIX launcher used a raw bash `case` against the JSON-encoded value, so paths containing characters that JSON escapes (e.g. /tmp/back\slash, /tmp/O"Brien) caused the launcher to reject its own healthy backend. - /api/health is unauthenticated and Studio supports `-H 0.0.0.0`, so any reachable client could read the absolute install path (username, home dir, workspace name, CI checkout path). - The verification was gated on $UNSLOTH_STUDIO_HOME being set at runtime, so a default-mode launcher would attach to a sibling env-mode Studio listening on the same port instead of starting its own. The fix replaces the raw path with a SHA-256 hex digest computed at install time and baked into the generated launcher (mirroring how @@DATA_DIR@@ is substituted today): studio/backend/main.py: /api/health now returns `studio_root_id = sha256(str(_studio_root()))` instead of the raw `studio_root` path. install.sh: computes `_css_studio_root_id` once from $STUDIO_HOME using python3, bakes `_EXPECTED_STUDIO_ROOT_ID='@@STUDIO_ROOT_ID@@'` into the launcher heredoc, and adds `s|@@STUDIO_ROOT_ID@@|...|g` to the existing sed pipeline for ALL modes (env / home / default). _check_health verifies the baked id substring-matches the JSON response. Hex-only so no shell or sed escape corner cases. install.ps1: same shape on Windows. SHA256 the $StudioHome bytes, lower hex, bake `$_ExpectedStudioRootId = '...'` into the launcher heredoc. Test-StudioHealth now compares `$resp.studio_root_id -eq $_ExpectedStudioRootId` unconditionally (no special-case for env-mode). Default-mode launchers also bake their expected id, so two coexisting Studio installs on the same machine can no longer cross-attach. * Studio: harden launcher root-id and split install-time mode from runtime env - install.sh launcher: compute studio_root_id with the venv Python (uv-managed systems may not have system python3) and canonicalize STUDIO_HOME with cd -P/pwd -P so default and home-redirect modes match the backend's Path(sys.prefix).resolve() canonicalization. Fail fast instead of silently baking an empty discriminator. - install.sh launcher heredoc: gate PORT_FILE / namespaced LOCK_DIR on a baked install-time mode flag (@@INSTALLED_IS_ENV_MODE@@) instead of the runtime UNSLOTH_STUDIO_HOME variable so a sourced custom-root studio.conf cannot flip a default-mode launcher into env-mode behavior with stale state. - studio/backend/main.py: cache the studio_root_id digest at module load so /api/health does not recompute hashlib + filesystem probes on every poll. - studio/backend/core/inference/llama_cpp.py: widen the studio_root() probe except clause from ImportError to (ImportError, OSError, ValueError) so it matches the sibling _kill_orphaned_servers handler and tolerates Path.resolve failures from broken symlinks or odd codecs. * Studio: align launcher root-id digest with backend canonicalization - studio/backend/main.py: hash the already-resolved _STUDIO_ROOT_RESOLVED instead of recomputing str(_studio_root()); the default fallback in storage_roots returns Path.home()/.unsloth/studio without .resolve(), so on systems where $HOME is a symlink (NFS / AFS / Docker) the cached digest now matches install.sh's cd -P/pwd -P canonicalization and the launcher no longer rejects its own healthy backend. - install.ps1: canonicalize $StudioHome via Resolve-Path before the SHA256 compute (env-mode already resolves at line 121, only default and profile branches were raw); a junctioned USERPROFILE now produces the same digest the backend computes via Path.resolve() for the same install. - install.sh launcher template: substitute the non-user-controlled @@STUDIO_ROOT_ID@@ and @@INSTALLED_IS_ENV_MODE@@ placeholders before the user-controlled @@DATA_DIR@@ pass so a $DATA_DIR that contains the literal placeholder text cannot be mutated by the second sed. * Studio: tighten installer rationale comments * Studio install: extend workspace-guard test coverage Add behavioral coverage for env-mode workspace guards across install.sh, install.ps1, studio/setup.sh, studio/setup.ps1, the launcher root-id discriminator, and the backend's /api/health response. Also refresh the custom-mode llama.cpp resilience assertion so it matches the implementation that intentionally excludes the legacy tree from search_roots. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Honor STUDIO_HOME alias, fix workspace-guard test harness, harden rollback The PR title and description promise STUDIO_HOME as a priority-2 alias to UNSLOTH_STUDIO_HOME, but the implementation only read the longer name in all six resolution sites. Wire the alias through install.sh, install.ps1, studio/setup.sh, studio/setup.ps1, the Python storage_roots resolver, and the unsloth_cli studio resolver. UNSLOTH_STUDIO_HOME wins when both are set (more specific signal beats the generic alias). Whitespace-only values are now treated as unset to match the Python resolvers' .strip() semantics, preventing install/runtime layout drift where the installer would create a literal " " directory while the backend fell through to the legacy default. Error messages and the substep status line report the env-var name the user actually set ("UNSLOTH_STUDIO_HOME=..." vs "STUDIO_HOME=...") so diagnostics stay accurate under either spelling. Test harness fix: tests/test_studio_install_workspace_guard.py extracted the install.sh venv-replacement block, but after the merge that block delegates to _start_studio_venv_replacement (defined further up in install.sh, not in the extracted snippet). Five sentinel-positive tests echoed RESULT=ok but never moved $VENV_DIR. Add a single _INSTALL_GUARD_STUBS constant that stands in a minimal mv-based stub plus a no-op substep, and route every inline test script through a new _build_install_guard_script() helper. All 50 tests now pass (was 45/50). Rollback hardening: Start-StudioVenvRollback / Restore-StudioVenvRollback / Complete-StudioVenvRollback in install.ps1 used plain Test-Path, Move-Item, Remove-Item against paths derived from $StudioHome. With a custom UNSLOTH_STUDIO_HOME containing brackets (the very motivation for the broader -LiteralPath sweep this PR set out to do), rollback would silently misbehave under wildcard interpretation, turning a recoverable install error into a destroyed env. Same fix for the --local Tauri overlay block (Test-Path / Copy-Item / Get-FileHash on $VenvDir-derived paths). * Replace studio_root_id path-hash with per-install opaque id The previous design computed studio_root_id as sha256 of the resolved $STUDIO_HOME path, both at install time (baked into the launcher) and at backend startup (returned via /api/health). This worked but had three weaknesses: 1. Information disclosure on -H 0.0.0.0: anyone reaching /api/health could confirm a guessed install path (username, workspace name, etc.) by replaying the same hash. 2. Canonicalization brittleness: launcher (cd -P/pwd -P) and backend (Path.resolve()) had to produce identical strings, which required careful symlink/junction handling on every site (cycles 17-27 of the PR review history were entirely about closing this drift). 3. Stale-launcher attach: an uninstall + reinstall at the same path produced the same hash, so a launcher from the previous install would silently attach to the new (incompatible) backend. Replace the path-hash with a per-install opaque id: - install.sh and install.ps1 generate 32 bytes from the platform CSPRNG (/dev/urandom on POSIX with a python3 secrets fallback; RandomNumberGenerator.Create().GetBytes on Windows) and persist it to $STUDIO_HOME/share/studio_install_id with mode 0600. Atomic temp-file-rename so a crash mid-install can't leave a half-written id. The check 'if [ ! -s "$_css_id_file" ]' / Test-Path makes generation idempotent across re-runs (so re-running install.sh doesn't invalidate previously-baked launchers in the same install root). - studio/backend/main.py replaces hashlib.sha256 with _read_studio_install_id(), which reads $STUDIO_HOME/share/studio_install_id once at module load. Validates the content against ^[0-9a-f]{64}$ so malformed/truncated/uppercase/wrong-length content returns "" and triggers the launcher's existing "no baked id, accept any healthy Unsloth backend" fallback path. - /api/health field name (studio_root_id) and wire format (64 hex chars) preserved for compatibility with launchers already shipped via earlier PR iterations. Tests: - Drop test_install_sh_root_id_matches_backend_resolved_under_symlinked_home and test_install_ps1_canonicalizes_studio_home_before_root_id_hash -- the entire reason these existed (cd -P/Resolve-Path/Path.resolve() digest agreement under symlinks/junctions) is moot when the id comes from a file rather than from the path. - Drop test_main_py_studio_root_id_hashes_resolved_root_not_unresolved (no more hashing). - Rewrite test_main_py_studio_root_id_caches_at_module_load to assert the file-read pattern; add test_main_py_read_studio_install_id_validates_hex_and_handles_missing to pin the exact rejection rules (empty / non-hex / wrong case / wrong length all -> ""). - Rewrite test_install_sh_create_shortcuts_uses_venv_python_first as test_install_sh_create_shortcuts_seeds_id_from_csprng_with_python_fallback with a behavioral subprocess check that re-invocation is idempotent. - Rename test_check_health_handles_path_with_backslash_via_hash to test_check_health_handles_arbitrary_id_token (the JSON-escape concern it pinned is preserved -- ids are hex-only by construction -- but the test no longer derives the id from a path). - Add test_install_sh_install_id_survives_symlinked_studio_home as a regression test pinning that the new design has zero canonicalization drift across symlinked parents. - Update test_install_sh_bakes_studio_root_id_into_launcher and test_install_ps1_bakes_studio_root_id_into_launcher to assert the CSPRNG seed and the file location. 49/49 tests pass. Behavioral verification: install.sh-style generation is idempotent across runs, three parallel installs at different roots get distinct ids, reinstall at the same path produces a new id (so stale launchers correctly fail to attach to the new backend), and symlinked-\$HOME no longer causes launcher/backend disagreement. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Daniel Han <unslothai@gmail.com>	2026-05-05 23:17:40 -07:00
Daniel Han	f0d03655e8	Studio: add folder browser modal for Custom Folders (#5035) ... * Studio: add folder browser modal for Custom Folders The Custom Folders row in the model picker currently only accepts a typed path. On a remote-served Studio (Colab, shared workstation) that means the user has to guess or paste the exact server-side absolute path. A native browser folder picker can't solve this: HTML `<input type="file" webkitdirectory>` hides the absolute path for security, and the File System Access API (Chrome/Edge only) returns handles rather than strings, neither of which the server can act on. This PR adds a small in-app directory browser that lists paths on the server and hands the chosen string back to the existing `POST /api/models/scan-folders` flow. ## Backend * New endpoint `GET /api/models/browse-folders`: * `path` query param (expands `~`, accepts relative or absolute; empty defaults to the user's home directory). * `show_hidden` boolean to include dotfiles/dotdirs. * Returns `{current, parent, entries[], suggestions[]}`. `parent` is null at the filesystem root. * Immediate subdirectories only (no recursion); files are never returned. * `entries[].has_models` is a cheap hint: the directory looks like it holds models if it is named `models--*` (HF hub cache layout) or one of the first 64 children is a .gguf/.safetensors/config.json/ adapter_config.json or another `models--*` subfolder. * Sort order: model-bearing dirs, then plain, then hidden; case- insensitive alphabetical within each bucket. * Suggestions auto-populate from HOME, the HF cache root, and any already-registered scan folders, deduplicated. * Error surface: 404 for missing path, 400 for non-directory, 403 on permission errors. Auth-required like the other models routes. * New Pydantic schemas `BrowseEntry` and `BrowseFoldersResponse` in `studio/backend/models/models.py`. ## Frontend * New `FolderBrowser` component (`studio/frontend/src/components/assistant-ui/model-selector/folder-browser.tsx`) using the existing `Dialog` primitive. Features: * Clickable breadcrumb with a `..` row for parent navigation. * Quick-pick chips for the server-provided suggestions. * `Show hidden` checkbox. * In-flight fetch cancellation via AbortController so rapid navigation doesn't flash stale results. * Badges model-bearing directories inline. * `chat-api.ts` gains `browseFolders(path?, showHidden?)` and matching types. * `pickers.tsx` adds a folder-magnifier icon next to the existing `Add` button. Opening the browser seeds it with whatever the user has already typed; confirming fills the text input, leaving the existing validation and save flow unchanged. ## What it does NOT change * The existing text-input flow still works; the browser is additive. * No new permissions or escalation; the endpoint reads only directories the server process is already allowed to read. * No model scanning or filesystem mutation happens from the browser itself -- it just returns basenames for render. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: cap folder-browser entries and expose truncated flag Pointing the folder browser at a huge directory (``/usr/lib``, ``/proc``, or a synthetic tree with thousands of subfolders) previously walked the whole listing and stat-probed every child via ``_looks_like_model_dir``. That is both a DoS shape for the server process and a large-payload surprise for the client. Introduce a hard cap of 2000 subdirectory entries and a ``truncated: bool`` field on the response. The frontend renders a small hint below the list when it fires, prompting the user to narrow the path. Below-cap directories are unchanged. Verified end-to-end against the live backend with a synthetic tree of 2050 directories: response lands at 2000 entries, ``truncated=true``, listing finishes in sub-second time (versus tens of seconds if we were stat-storming). * Studio: suggest LM Studio / Ollama dirs + 2-level model probe Three improvements to the folder-browser, driven by actually dropping an LM Studio-style install (publisher/model/weights.gguf) into the sandbox and walking the UX: ## 1. Quick-pick chips for other local-LLM tools `well_known_model_dirs()` (new) returns paths commonly used by adjacent tools. Only paths that exist are returned so the UI never shows dead chips. * LM Studio current + legacy roots + user-configured `downloadsFolder` from its `settings.json` (reuses the existing `lmstudio_model_dirs()` helper). * Ollama: `$OLLAMA_MODELS` env override, then `~/.ollama/models`, `/usr/share/ollama/.ollama/models`, and `/var/lib/ollama/.ollama/models` (the systemd-service install path surfaced in the upstream "where is everything?" issue). * Generic user-choice locations: `~/models`, `~/Models`. Dedup is stable across all sources. ## 2. Two-level model-bearing probe LM Studio and Ollama both use `root/publisher/model/weights.gguf`. The previous `has_models` heuristic only probed one level, so the publisher dir (whose immediate children are model dirs, not weight files) was always marked as non-model-bearing. Pulled the direct- signal logic into `_has_direct_model_signal` and added a grandchild probe so the classic layout is now recognised. Still O(PROBE^2) worst-case, still returns immediately for `models--*` names (HF cache layout) and for any direct weight file. ## 3. model_files_here hint on response body A leaf model dir (just GGUFs, no subdirs) previously rendered as `(empty directory)` in the modal, confusing users into thinking the folder wasn't scannable. Added a `model_files_here` count on the response (capped at 200) and a small hint row in the modal: `N model files in this folder. Click "Use this folder" to scan it.` ## Verification Simulated an LM Studio install by downloading the real 84 MB `unsloth/SmolLM2-135M-Instruct-Q2_K.gguf` into `~/.lmstudio/models/unsloth/SmolLM2-135M-Instruct-GGUF/`. Confirmed end-to-end: * Home listing suggests `~/.lmstudio/models` as a chip. * Browsing `~/.lmstudio/models` flags `unsloth` (publisher) as `has_models=true` via the 2-level probe. * Browsing the publisher flags `SmolLM2-135M-Instruct-GGUF` (model dir) as `has_models=true`. * Browsing the model dir returns empty entries but `model_files_here=1`, and the frontend renders a hint telling the user it is a valid target. * Studio: one-click scan-folder add + prominent remove + plain search icon Three small Custom Folders UX fixes after real-use walkthrough: * **One-click add from the folder browser**. Confirming `Use this folder` now submits the path directly to `POST /api/models/scan-folders` instead of just populating the text input. `handleAddFolder` takes an optional explicit path so the submit lands in the same tick as `setFolderInput`, avoiding a state-flush race. The typed-path + `Add` button flow is unchanged. * **Prominent remove X on scan folders**. The per-folder delete button was `text-muted-foreground/40` and hidden entirely on desktop until hovered (`md:opacity-0 md:group-hover:opacity-100`). Dropped the hover-only cloak, bumped color to `text-foreground/70`, added a red hover/focus background, and sized the icon up from `size-2.5` to `size-3`. Always visible on every viewport. * **Plain search icon for the Browse button**. `FolderSearchIcon` replaced with `Search01Icon` so it reads as a simple "find a folder" action alongside the existing `Add01Icon`. * Studio: align Custom Folders + and X buttons on the same right edge The Custom Folders header used `px-2.5` with a `p-0.5` icon button, while each folder row used `px-3` with a `p-1` button. That put the X icon 4px further from the right edge than the +. Normalised both rows to `px-2.5` with `p-1` so the two icons share a column. * Studio: empty-state button opens the folder browser directly The first-run empty state for Custom Folders was a text link reading "+ Add a folder to scan for local models" whose click toggled the text input. That's the wrong default: a user hitting the empty state usually doesn't know what absolute path to type, which is exactly what the folder browser is for. * Reword to "Browse for a models folder" with a search-icon affordance so the label matches what the click does. * Click opens the folder browser modal directly. The typed-path + Add button flow is still available via the + icon in the section header, so users who know their path keep that option. * Slightly bump the muted foreground opacity (70 -> hover:foreground) so the button reads as a primary empty-state action rather than a throwaway hint. * Studio: Custom Folders header gets a dedicated search + add button pair The Custom Folders section header had a single toggle button that flipped between + and X. That put the folder-browser entry point behind the separate empty-state link. Cleaner layout: two buttons in the header, search first, then add. * Search icon (left) opens the folder browser modal directly. * Plus icon (right) toggles the text-path input (unchanged). * The first-run empty-state link is removed -- the two header icons cover both flows on every state. Both buttons share the same padding / icon size so they line up with each other and with the per-folder remove X. * Studio: sandbox folder browser + bound caps + UX recoveries PR review fixes for the Custom Folders folder browser. Closes the high-severity CodeQL path-traversal alert and addresses the codex / gemini P2 findings. Backend (studio/backend/routes/models.py): * New _build_browse_allowlist + _is_path_inside_allowlist sandbox. browse_folders now refuses any target that doesn't resolve under HOME, HF cache, Studio dirs, registered scan folders, or the well-known third-party model dirs. realpath() is used so symlink traversal cannot escape the sandbox. Also gates the parent crumb so the up-row hides instead of 403'ing. * _BROWSE_ENTRY_CAP now bounds *visited* iterdir entries, not *appended* entries. Dirs full of files (or hidden subdirs when show_hidden is False) used to defeat the cap. * _count_model_files gets the same visited-count fix. * PermissionError no longer swallowed silently inside the enumeration / counter loops -- now logged at debug. Frontend (folder-browser.tsx, pickers.tsx, chat-api.ts): * splitBreadcrumb stops mangling literal backslashes inside POSIX filenames; only Windows-style absolute paths trigger separator normalization. The Windows drive crumb value is now C:/ (drive root) instead of C: (drive-relative CWD-on-C). * browseFolders accepts and forwards an AbortSignal so cancelled navigations actually cancel the in-flight backend enumeration. * On initial-path fetch error, FolderBrowser now falls back to HOME instead of leaving the modal as an empty dead end. * When the auto-add path (one-click "Use this folder") fails, the failure now surfaces via toast in addition to the inline paragraph (which is hidden when the typed-input panel is closed). * Studio: rebuild browse target from trusted root for CodeQL clean dataflow CodeQL's py/path-injection rule kept flagging the post-validation filesystem operations because the sandbox check lived inside a helper function (_is_path_inside_allowlist) and CodeQL only does intra-procedural taint tracking by default. The user-derived ``target`` was still flowing into ``target.exists`` / ``target.is_dir`` / ``target.iterdir``. The fix: after resolving the user-supplied ``candidate_path``, locate the matching trusted root from the allowlist and rebuild ``target`` by appending each individually-validated segment to that trusted root. Each segment is rejected if it isn't a single safe path component (no separators, no ``..``, no empty/dot). The downstream filesystem ops now operate on a Path constructed entirely from ``allowed_roots`` (trusted) plus those validated segments, so CodeQL's dataflow no longer sees a tainted source. Behavior is unchanged for all valid inputs -- only the construction of ``target`` is restructured. Live + unit tests all pass (58 selected, 7 deselected for Playwright env). * Studio: walk browse paths from trusted roots for CodeQL --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Ubuntu <ubuntu@h100-8-cheapest.us-east5-a.c.unsloth.internal>	2026-04-15 08:04:33 -07:00
Lee Jackson	a29b4e23fd	studio: reuse HF cached repo casing to prevent duplicate downloads (#4822) ... * fix(studio): reuse HF cached repo casing to prevent duplicate downloads * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Move cache case resolution tests to separate PR Tests for resolve_cached_repo_id_case and get_model_config case resolution belong in their own PR to keep this change focused on the runtime fix. * fix(studio): debug-log HF_HUB_CACHE fallback in path_utils * Fix stale memoization in resolve_cached_repo_id_case - Check exact-case path before memo to ensure a newly-appeared exact match always wins over a previously memoized variant - Validate memoized entries still exist on disk before returning them to prevent stale results when cache dirs are deleted/recreated * Minor cleanups for cache case resolution - Use .is_dir() instead of .exists() for exact-case cache check (cache entries are always directories) - Remove redundant fallback in _detect_audio_from_tokenizer since get_cache_path already handles case resolution and returns None when the model is not cached --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Daniel Han <danielhanchen@gmail.com>	2026-04-03 05:48:24 -07:00
Daniel Han	82d14b44d3	fix: preserve Windows drive-letter paths on native Windows (#4665) ... normalize_path() unconditionally converted Windows paths like C:\Users\... to WSL format /mnt/c/Users/..., which breaks path resolution on native Windows. This caused LM Studio GGUF models to fail detection (detect_gguf_model returned None for the invalid path), falling through to the Unsloth import path which requires a GPU. Now only performs the /mnt/ mapping when actually running under WSL. On native Windows, drive letters are preserved and backslashes are normalized to forward slashes.	2026-03-27 08:19:41 -07:00
Roland Tannous	562e54fc6e	Fix HF cache default and show LM Studio models in chat/inference (#4653) ... * fix: default HF cache to standard platform path instead of legacy Unsloth cache * feat: show LM Studio and local models in chat Fine-tuned tab * feat: show LM Studio models in Hub models tab * fix: fetch local models after auth refresh completes * Revert "fix: fetch local models after auth refresh completes" This reverts commit cfd61f0ac7. * fix: increase llama-server health check timeout to 600s for large models * feat: expandable GGUF variant picker for LM Studio local models * fix: show GGUF variant label for locally loaded LM Studio models * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix: show publisher name in LM Studio model labels * fix: set model_id for loose GGUF files in LM Studio publisher dirs * fix: show publisher prefix in Fine-tuned tab LM Studio models * fix: only use model_id for lmstudio source models * fix: only show LM Studio models in Hub tab on Mac/chat-only mode * fix: respect XDG_CACHE_HOME, handle Windows paths in isLocalPath, refresh LM Studio on remount - _setup_cache_env now reads XDG_CACHE_HOME (falls back to ~/.cache) instead of hard-coding ~/.cache/huggingface. This follows the standard HF cache resolution chain and respects distro/container overrides. - isLocalPath in GgufVariantExpander uses a regex that covers Windows drive letters (C:\, D:/), UNC paths (\\server\share), relative paths (./, ../), and tilde (~/) -- not just startsWith("/"). - HubModelPicker.useEffect now calls listLocalModels() before the alreadyCached early-return gate so LM Studio models are always refreshed on remount. Also seeds useState from _lmStudioCache for instant display on re-open. * fix: add comment explaining isLocalPath regex for Windows/cross-platform paths * fix: prioritize unsloth publisher in LM Studio model list * fix: scope unsloth-first sort to LM Studio models on all platforms * fix: add missing _lmStudioCache module-level declaration * fix: prioritize unsloth publisher before timestamp sort in LM Studio group --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Daniel Han <danielhanchen@gmail.com>	2026-03-27 06:59:27 -07:00
Roland Tannous	48a7884584	feat: multi-source model discovery (HF default, legacy cache, LM Studio) (#4591) ... * feat: multi-source model discovery (HF default, legacy cache, LM Studio) * Fix multi-source model discovery bugs - Fix lmstudio_model_dirs: add ~/.lmstudio/models as default path, remove dead sys.platform branch, add dedup via seen set - Fix _setup_cache_env: preserve legacy HF cache env vars when the legacy hub directory exists and is non-empty - Fix _scan_lmstudio_dir: use absolute path for id field so is_local_path() returns True - Remove LM Studio dirs from allowed_roots (scanned unconditionally) - Replace bare except passes with logger.warning in legacy cache blocks - Fix delete_cached_model to search both default and legacy HF caches - Make lmstudio_dirs non-optional in TS interface (matches Python schema) - Exclude lmstudio source from trainable model filter - Remove unused import sys * Scan HF default cache alongside legacy and active caches When _setup_cache_env overrides HF_HUB_CACHE to the legacy Unsloth path, the standard HF default cache (~/.cache/huggingface/hub) was never scanned, hiding models downloaded before Unsloth Studio was installed. Add hf_default_cache_dir() and _all_hf_cache_scans() helper that deduplicates and scans all three HF cache locations (active, legacy, default). Used in list_local_models, list_cached_gguf, list_cached_models, and delete_cached_model. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: Daniel Han <danielhanchen@gmail.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-03-25 07:48:04 -07:00
Roland Tannous	1f498a73e6	Revert "feat: multi-source model discovery (HF default, legacy cache, LM Studio)" ... This reverts commit d56b115bb4.	2026-03-25 13:35:03 +00:00
Roland Tannous	d56b115bb4	feat: multi-source model discovery (HF default, legacy cache, LM Studio)	2026-03-25 13:24:46 +00:00
Wasim Yousef Said	208862218d	feat(studio): training history persistence and past runs viewer (#4501) ... * feat(db): add SQLite storage layer for training history * feat(api): add training history endpoints and response models * feat(training): integrate DB persistence into training event loop * feat(ui): add training history views and card grid * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix(studio): address review issues in training history persistence - Strip hf_token/wandb_token from config before SQLite storage - Add UUID suffix to job_id for collision resistance - Use isfinite() for 0.0 metric handling throughout - Respect _should_stop in error event finalization - Run schema DDL once per process, not per connection - Close connection on schema init failure - Guard cleanup_orphaned_runs at startup - Cap _metric_buffer at 500 entries - Make FLUSH_THRESHOLD a class constant - Map 'running' to 'training' phase in historical view - Derive LR/GradNorm from history arrays in historical view - Fix nested button with div[role=button] in history cards - Guard String(value) against null/undefined in config popover - Clear selectedHistoryRunId on auto tab switch * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix(studio): address round-2 review findings across training backend and frontend Backend (training.py): - Move state mutation after proc.start() so a failed spawn does not wedge the backend with is_training=True - Create DB run row eagerly after proc.start() so runs appear in history during model loading, not after first metric event - Rewrite _flush_metrics_to_db() with snapshot-before-insert pattern to preserve metrics arriving during the write and retain buffer on failure - Guard eval_loss with float() coercion and math.isfinite(), matching the existing grad_norm guard - Increase pump thread join timeout from 3s to 8s to cover SQLite's default 5s lock timeout Frontend (studio-page.tsx): - Fix history navigation: check isTrainingRunning instead of showTrainingView in onSelectRun so completed runs are not misrouted - Replace activeTab state + auto-switch useEffect with derived tab to eliminate react-hooks/set-state-in-effect lint violation Frontend (historical-training-view.tsx): - Add explicit "running" branch to message ternary so running runs no longer fall through to "Training errored" - Derive loading from detail/error state and move cleanup to effect return to eliminate react-hooks/set-state-in-effect lint violation Frontend (progress-section.tsx): - Derive stopRequested from isTrainingRunning && stopRequestedLocal to eliminate react-hooks/set-state-in-effect lint violation and remove unused useEffect import * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix(studio): resolve 3 remaining bugs from round-2 review 1. Stuck on Current Run tab [12/20]: Only force "current-run" tab when isTrainingRunning is true, not when stale completed-run data exists. After training ends, users can freely navigate to Configure. 2. Incomplete metric sanitization [7/20]: Apply float() coercion and isfinite() guards to loss and learning_rate, matching the existing pattern used by grad_norm and eval_loss. Prevents TypeError from string values and NaN leaks into history arrays. 3. Stop button state leak across runs [10/20]: Add key={runtime.jobId} to ProgressSection so React remounts it when a new run starts, resetting stopRequestedLocal state. * fix(studio): deduplicate loss/lr sanitization in training event handler Reuse _safe_loss/_safe_lr from the progress update block instead of re-sanitizing the same raw event values for metric history. * fix(studio): restore loss > 0 guard to prevent eval steps injecting 0.0 into metric histories Round-2/3 fixes relaxed the history append guard from `loss > 0` to `loss is not None`, which let eval-only log events (where loss defaults to 0.0) append fake zeros into loss_history and lr_history. Restore the `loss > 0` check to match the worker's own has_train_loss gate. The float() coercion and isfinite() sanitization from round-3 remain intact. * fix(studio): resolve training history bugs — nullable loss/lr, tab nav, sparkline * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Daniel Han <danielhanchen@gmail.com>	2026-03-25 00:58:55 -07:00
Wasim Yousef Said	dd283b0605	feat(studio): multi-file unstructured seed upload with better backend extraction (#4468) ... * fix(recipe-studio): prevent fitView from zooming to wrong location on recipe load * feat: add pymupdf/python-docx deps and unstructured uploads storage root * feat: add POST /seed/upload-unstructured-file endpoint * feat: add multi-file chunking with source_file column * feat: update frontend types and API layer for multi-file upload * feat: round-robin preview rows across source files Ensures every uploaded file is represented in the preview table by cycling through sources instead of just taking the first N rows. * fix: disable OCR, fix auto-load timing, fix persistence on reload - Disable pymupdf4llm OCR with write_images=False, show_progress=False - Replace onAllUploaded callback with useEffect that detects uploading→done transition (avoids stale closure reading empty file IDs) - Fix importer to preserve file IDs from saved recipes instead of clearing (clearing only happens at share time via sanitizeSeedForShare) * fix: harden unstructured upload with input validation and state fixes Validate block_id/file_id with alphanumeric regex to prevent path traversal, use exact stem match for file deletion, add error handling for metadata writes and empty files, fix React stale closures and object mutations in upload loop, and correct validation logic for unstructured seed resolved_paths. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix: address PR review - legacy path import, share sanitizer, sync effect Promote legacy source.path into resolved_paths for old unstructured recipes, clear source.paths in share sanitizer to prevent leaking local filesystem paths, and gate file sync effect to dialog open transition so users can actually delete all uploaded files. * fix: CSV column fix (BOM + whitespace + unnamed index re-save) for #4470 * fix: harden unstructured upload flow and polish dialog UX * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-03-20 13:22:42 -07:00
pre-commit-ci[bot]	f4fbbcaec8	[pre-commit.ci] auto fixes from pre-commit.com hooks ... for more information, see https://pre-commit.ci	2026-03-15 05:24:06 -07:00
Daniel Han	f4f69f16a6	studio: centralize cache directory for all downloads ... Set HF_HOME, HF_HUB_CACHE, HF_XET_CACHE, UV_CACHE_DIR, and VLLM_CACHE_ROOT to a unified location under ~/.unsloth/studio/cache/ on startup. This keeps all model downloads, datasets, and caches in one place instead of scattered across ~/.cache/huggingface, ~/.cache/uv, etc. Layout: ~/.unsloth/studio/cache/ huggingface/ (HF_HOME) hub/ (HF_HUB_CACHE -- model/dataset downloads) xet/ (HF_XET_CACHE -- xet blob store) uv/ (UV_CACHE_DIR -- uv package cache) vllm/ (VLLM_CACHE_ROOT -- vllm compiled kernels) Only sets variables that are not already in the environment, so user overrides (e.g. HF_HOME=/data/models) are respected. Cross-platform: uses Path.home() which resolves correctly on Linux (~), macOS (~), and Windows (C:\Users\<user>).	2026-03-15 05:24:06 -07:00
Roland Tannous	47654cb91c	Final cleanup	2026-03-12 18:28:04 +00:00
Roland Tannous	a2baf80511	Update license headers	2026-03-12 17:23:10 +00:00
Shine1i	bbb4cd0f0b	feat(studio): add auth-specific paths and integrate auth database location	2026-03-11 20:19:52 +00:00
Shine1i	7012b8396f	fix(studio): update temporary directory path to use system temp dir	2026-03-11 20:19:52 +00:00
Shine1i	904e440513	feat(studio): studio storage roots path utilities	2026-03-11 20:19:52 +00:00
Roland Tannous	817f2e8dcc	feat: integrate structlog, configure workers for prod logging, and migrate print statements	2026-03-11 12:33:16 +00:00
Roland Tannous	daa50d0756	Revert "Merge pull request #347 from unslothai/feature/studio-storage-roots" ... This reverts commit 6b43e33ff1, reversing changes made to 9edadaf21f.	2026-03-10 01:52:47 +00:00
Shine1i	109db14817	feat(studio): add auth-specific paths and integrate auth database location	2026-03-09 23:48:31 +00:00
Shine1i	958bdef43e	fix(studio): update temporary directory path to use system temp dir	2026-03-09 23:48:31 +00:00
Shine1i	5301514775	feat(studio): studio storage roots path utilities	2026-03-09 23:48:31 +00:00
Roland Tannous	d882678fe4	Add AGPL-3.0 SPDX headers to all source files	2026-03-09 20:17:45 +00:00
Shine1i	8529f89a75	fix lora: outputs path local	2026-02-15 16:58:24 +01:00
Roland Tannous	544d6944d1	root studio folder	2026-02-02 09:13:49 +00:00