mirror of
https://github.com/unslothai/unsloth.git
synced 2026-08-25 08:42:25 +00:00
* Studio: quiet the liveness poll and keep access lines out of tauri.log Measured on an idle 4h desktop session, tauri.log was 5308 lines / 1.14 MB, of which 4056 (76%) were request_completed access records and 3355 (63%) came from six polling endpoints. At ~216 bytes a line the 5 MiB cap fills in about 18 hours, so a failure from the day before had already rotated out by the time anyone asked for it. /api/liveness is the desktop watchdog's own probe, roughly every 19s, and was in no suppressor at all: 760 lines, 14% of the file, saying the process is still up. Its sibling /api/health was already quiet. Put it in _QUIET_POLL_PATHS so it heartbeats; non-2xx still logs every time, which is the case that matters for a watchdog. Every backend stdout line is written to disk three times: the backend's own session log, the diagnostics phase log (stream-tagged and stamped), and tauri.log. The access records carry nothing in the third copy that the first two lack, so log them at debug!, below the file target's INFO filter. Non-JSON output -- the startup banner, "Hardware detected:", stderr, tracebacks -- keeps INFO, which is what anyone opening tauri.log is there for. The phase log still gets the unfiltered stream, so the support report loses nothing. Also: - Collapse carriage-return progress frames. Reading to \n means a tqdm bar arrives as every frame it drew concatenated onto one line; a single "Loading weights" bar measured 5086 bytes across three sinks. Keep the final frame, which is the one showing 100% and the elapsed total. Cap what reaches tauri.log at 16 KiB, the limit the phase log already used. - Bound logs/llama-server and logs/diffusion-server. Both directories are created per model load and were never pruned (319 files going back two months here). The server session log has capped itself since it was added, so lift that into utils/log_retention.prune_log_dir and call it from all three rather than write the loop a third time. - Log the top-models catalog at debug. Two lists of 40 repo names, one line each, on every boot to say a fetch succeeded; the counts say that. - Configure structlog in run.py before the first line it logs. main.py configures on import, several seconds later, so "run_server startup begin" rendered through the defaults (console, local time) and everything after it as JSON in UTC -- the clock appeared to jump hours between the first and second line of one stream. Not included: the same carriage-return collapse on the Python tee. Doing it there means line-buffering the file copy, which would withhold a partial line from server-*.log exactly when the server hangs mid-write. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Collapse progress frames in the session log too, and quiet remote-access The first pass collapsed carriage-return redraws on the Rust side only. It left the Python tee alone because doing it there seemed to mean buffering the file copy until a newline, which would withhold a partial line from server-*.log exactly when the server hangs mid-write. It does not have to. Hold back a frame, never a line: an unterminated chunk containing "\r" is a redraw and is kept until the next one supersedes it, while an unterminated chunk without one -- a prompt, or a traceback torn by a hang -- is written the moment it arrives. So the session log gets one line per bar instead of one per frame, and nothing diagnostic waits on a newline that may never come. The console side is untouched: it still sees every frame, which is the whole point of a progress bar. Also adds /api/settings/remote-access to the quiet polls (116 lines in the same 4h sample), for the same reason as the others: it is a plain read of a toggle, re-read whenever the settings dialog is open. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Install the tee before structlog, and stop three suppressors overreaching Four review findings, all real. structlog was configured at import, and PrintLoggerFactory snapshots sys.stdout while cache_logger_on_first_use freezes that snapshot into any logger that has already logged. run_server's own logger logs at line one and the tee is not installed until 45 lines later, so every subsequent run.py line -- the orphan sweep warnings, the whole rest of startup -- was pinned to the console and never reached the session log. Configure inside run_server instead, after the tee: the ordering bug the change set out to fix (line one rendering through structlog's defaults while line two rendered as JSON in UTC) is still fixed, and the file gets everything from line one. Moving the tee up also means the orphan sweep now lands on disk, where it was console-only before. A withheld progress frame was concatenated onto whatever came next, so a structlog record arriving mid-redraw landed as Loading 47%{"event": ...} and stopped being parseable. Close the frame off on its own line unless the next write is its own redraw or its terminator, and land a frame still held at close() rather than dropping it. The desktop watchdog polls every 15s (HEALTH_WATCHDOG_INTERVAL) plus up to a 10s probe budget, so _QUIET_POLL_DEDUP_MS at 10s could never close over two probes and putting /api/liveness in _QUIET_POLL_PATHS collapsed nothing. Give it its own 60s window, zeroed alongside the quiet window so --verbose still logs every probe, and pin the floor with a test that reads both constants out of commands.rs. is_backend_access_log_line matched on the event name alone, so it also dropped non-2xx records to debug! -- below tauri.log's INFO filter. Those are the ones the backend's heartbeat suppressor deliberately exempts, and a watchdog going red is what tauri.log is read for. Match only 2xx; a record with no status_code keeps its INFO line too. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * run.py: keep CRLF lines, stop leaking the tee, import LogConfig lazily Three defects in the same file, all found while checking this branch against main. Tests for each alongside. 1. The tee destroyed every CRLF-terminated line. _last_frame took rpartition("\r")[2] unconditionally, so the "\r" of a CRLF read as a redraw boundary whose final frame was empty, and the payload went with it: "Hardware detected: NVIDIA GeForce RTX 4090\r\n" -> "\n" "Traceback...\r\n File \"run.py\"...\r\nRuntimeError\r\n" -> "\n\n\n" On Windows that is every line a child process relays, so a relayed traceback landed in the session log as blank lines. It is also exactly the artifact #8690 added a UI for. The desktop reader already gets this right: trim_line_endings strips every trailing \r and \n before collapse_progress_frames sees the text. Mirror it here so the two sinks stay interchangeable for a reader, and fall back to the last non-blank frame the same way, so a bar that signs off with a bare \r keeps its 100% frame instead of writing a blank line. Also: a "\r" must never reach the file. The handle appends the platform terminator itself, so one that survives lands as "\r\r\n" on Windows. 2. A zero-length write reopened the record corruption fixed on 08-18. `data[:1] not in ("", "\r", "\n")` read write("") as a continuation of the held frame, so the close-off was skipped, the frame fell through to the unterminated branch, and the next record was glued onto it: ["\rLoading weights: 47%", "", '{"event": "model_loaded"}\n'] -> 'Loading weights: 47%{"event": "model_loaded"}' print("", end = "") is enough to reach it. Return early on empty data instead. 3. A rejected flag combination left the streams swapped. --secure --no-cloudflare is a deterministic preflight failure, but it now runs after _setup_server_disk_logging() has replaced sys.stdout and sys.stderr and opened a log handle, and nothing unwinds either. An embedder that catches the SystemExit keeps the tee, and its next run_server() call nests a second one, so every line is written twice. main does not do this. Validate before the tee. 4. `from loggers.config import LogConfig` at module scope broke Repo tests (CPU). 15 failures in tests/studio/install/test_selection_logic.py, all "No module named 'loggers.config'; 'loggers' is not a package". load_studio_run_module stands a bare types.ModuleType in for `loggers` before exec'ing run.py; a bare module has no __path__, so the import fails during collection and takes every test in the file with it. That fixture's own comment names the hazard. LogConfig has one use, inside run_server, so import it there rather than widening a stub three directories away. * Prune the log directories after opening the new file, and protect it Two problems with prune_log_dir as it stands. The cap never held. All three call sites prune before creating the log, so with keep=20 the directory settles at 21 and every load repeats the prune-to-20, create-the-21st cycle. Measured with 25 seeded files and 10 loads: 21 with this ordering, 20 when the prune runs after the open. And one odd directory entry disabled retention entirely. The mtime sort ran inside a single try/except, so a dangling symlink matching the glob raised on stat() and the whole call returned having deleted nothing. Seeded with 30 files plus one broken link, keep=5 left all 30. That is the unbounded growth this helper exists to stop, reintroduced by one bad name. Stat failures are now per-entry. Only regular files count now, too: a directory that happens to match the glob was taking one of the kept slots (25 files plus one such directory, keep=5, left 4 real logs). prune_log_dir grows a `protect` argument, and the call sites move to after the open. `protect` is never deleted and counts as one of the `keep`, so the total stays at keep. It is not enough to trust the new file to sort newest: two loads in the same second, or a clock that steps back, and it does not. The retry call site benefits most. The comment right above it explains that the -try{n} suffix exists so a respawn within the same second cannot truncate the crash log a retry warning just pointed the user at; pruning before the open on every attempt worked against that. New tests cover keep values, the protected file counting toward the cap and surviving even when it sorts oldest, dangling symlinks, symlinks pointing outside the directory, directories matching the glob, identical mtimes, family isolation, and a writer appending across a prune. * process.rs: CRLF and marker tests, and say what debug! means here collapse_progress_frames is correct on CRLF because trim_line_endings runs first, but nothing pinned that, and the Python tee that writes the backend's own session log got it wrong. Add the cases that hold the two implementations together: a CRLF payload keeps its text, a CRLF-terminated bar still collapses, a CRLF JSON record stays parseable and still classifies, a TAURI_PORT marker survives the collapse whether or not it shares its line with a redraw, and an all-blank line never keeps a carriage return. Plus the 16 KiB truncation landing on a character boundary for multi-byte text. The fallback for an all-blank line returned the whole text, which still holds the \r the caller asked to collapse away. Return the last frame instead. Two comment fixes. The new test module had been inserted between the C:\Windows\system32 comment and the managed_cli_working_dir_tests module it describes; move it back. And record what debug! actually does for a matched record: setup_logging in main.rs builds every logger at LevelFilter::Info and there is no RUST_LOG or any other way to raise it, so log::max_level() is Info and a matched record is dropped outright rather than demoted, including under backend --verbose, which makes it emit more of them. That is fine, but it is not obvious from the call site, and the surrounding comment argued the opposite case for non-2xx. The records are still written verbatim to the diagnostics phase log (append_phase_line runs before this decision) and to the backend session log, both of which the support report and Settings > Logs already offer. Separately, test_watchdog_window_outlasts_the_probe_interval read the module global, so it failed for anyone with UNSLOTH_STUDIO_ACCESS_LOG_WATCHDOG_DEDUP_MS exported; read the default instead. Its two regex lookups now report what moved in commands.rs rather than raising AttributeError on .group(1). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Tighten the comments this PR adds Comment-only pass over the 10 files the PR touches. No code change: the AST gate (comment_tools.py check --strip-docstrings) reports 9/9 files comments-only, the Rust file has no non-comment line in the diff, and no docstring was dropped. 114 insertions, 142 deletions. Heaviest in run.py and process.rs, where the same explanation was being given twice, once in a doc comment and again at the call site. Every load-bearing why is kept: the structlog factory/cache ordering, why a CRLF terminator is not a redraw, why the watchdog window has to outlast the poll, why prune runs after the open, and what debug! means for a matched record. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Daniel Han <danielhanchen@gmail.com>
178 lines
6.3 KiB
Python
178 lines
6.3 KiB
Python
# SPDX-License-Identifier: AGPL-3.0-only
|
|
# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved. See /studio/LICENSE.AGPL-3.0
|
|
|
|
"""The session log's copy of carriage-return progress bars.
|
|
|
|
A terminal overwrites a redraw in place; a file keeps every frame, so one tqdm bar landed as
|
|
kilobytes of near-identical text. The tee keeps the last frame only, and withholds nothing
|
|
except frames -- anything without a "\\r" is written the moment it arrives, so a hang cannot
|
|
swallow a partial traceback or a prompt.
|
|
"""
|
|
|
|
import io
|
|
import json
|
|
|
|
from run import _TeeStream
|
|
|
|
|
|
class _Sink:
|
|
def __init__(self):
|
|
self.buf = io.StringIO()
|
|
|
|
def write(self, data):
|
|
self.buf.write(data)
|
|
return len(data)
|
|
|
|
def flush(self):
|
|
pass
|
|
|
|
@property
|
|
def text(self):
|
|
return self.buf.getvalue()
|
|
|
|
|
|
def _tee(chunks):
|
|
log, console = _Sink(), _Sink()
|
|
stream = _TeeStream(console, log)
|
|
for chunk in chunks:
|
|
stream.write(chunk)
|
|
return log.text, console.text
|
|
|
|
|
|
def test_plain_output_is_unchanged_on_both_sides():
|
|
log, console = _tee(["plain line\n", "another\n"])
|
|
assert log == "plain line\nanother\n"
|
|
assert console == "plain line\nanother\n"
|
|
|
|
|
|
def test_console_always_sees_every_frame():
|
|
_log, console = _tee(["\rbar 1%", "\rbar 50%", "\rbar 100%", "\n"])
|
|
# The animation is the console's whole point; only the file copy collapses.
|
|
assert console == "\rbar 1%\rbar 50%\rbar 100%\n"
|
|
|
|
|
|
def test_progress_bar_collapses_to_its_final_frame():
|
|
log, _console = _tee(["\rbar 1%", "\rbar 50%", "\rbar 100%", "\n"])
|
|
assert log == "bar 100%\n"
|
|
|
|
|
|
def test_bar_between_real_lines_keeps_both():
|
|
log, _console = _tee(["Loading\n", "\ra 10%", "\ra 99%", "\n", "done\n"])
|
|
assert log == "Loading\na 99%\ndone\n"
|
|
|
|
|
|
def test_several_bars_in_one_chunk_collapse_per_line():
|
|
log, _console = _tee(["a\rb\rc\nd\re\n"])
|
|
assert log == "c\ne\n"
|
|
|
|
|
|
def test_unterminated_prompt_after_a_bar_is_not_withheld():
|
|
# "Start Unsloth Studio now? [Y/n]: " never gets a newline; it must still reach the file,
|
|
# and on its own line rather than glued to the frame that was being held.
|
|
log, _console = _tee(["\rbar 40%", "Start Unsloth Studio now? [Y/n]: "])
|
|
assert log == "bar 40%\nStart Unsloth Studio now? [Y/n]: "
|
|
|
|
|
|
def test_record_after_a_held_frame_stays_parseable():
|
|
# The reason the frame is closed off rather than prefixed: a structlog record arriving
|
|
# while a bar is mid-redraw must still be one JSON object on one line.
|
|
log, _console = _tee(["\rLoading weights: 47%", '{"event": "model_loaded"}\n'])
|
|
lines = log.splitlines()
|
|
assert lines == ["Loading weights: 47%", '{"event": "model_loaded"}']
|
|
json.loads(lines[-1])
|
|
|
|
|
|
def test_close_lands_a_frame_nothing_came_back_to_supersede():
|
|
log, console = _Sink(), _Sink()
|
|
stream = _TeeStream(console, log)
|
|
stream.write("\rbar 90%")
|
|
stream.close()
|
|
assert log.text == "bar 90%\n"
|
|
|
|
|
|
def test_hang_mid_bar_keeps_the_real_partial_line():
|
|
# The case that decides whether this is safe: a torn line is written, a frame is not.
|
|
log, _console = _tee(["Traceback (most recent call last):", "\rbar 5%"])
|
|
assert log == "Traceback (most recent call last):"
|
|
|
|
|
|
def test_file_failure_never_reaches_the_console():
|
|
class Exploding(_Sink):
|
|
def write(self, data):
|
|
raise OSError("disk full")
|
|
|
|
console = _Sink()
|
|
stream = _TeeStream(console, Exploding())
|
|
stream.write("still printed\n")
|
|
assert console.text == "still printed\n"
|
|
|
|
|
|
# ---------------------------------------------------------------------------------------
|
|
# A "\r" is only a redraw when something follows it on the same line.
|
|
# ---------------------------------------------------------------------------------------
|
|
|
|
|
|
def test_a_crlf_line_keeps_its_payload():
|
|
# "\r\n" is one terminator. Reading its "\r" as a redraw keeps the empty text after it
|
|
# and drops the line -- and on Windows every relayed child line arrives in this shape,
|
|
# so the session log goes blank exactly where the evidence should be.
|
|
log, _console = _tee(["Hardware detected: NVIDIA GeForce RTX 4090\r\n"])
|
|
assert log == "Hardware detected: NVIDIA GeForce RTX 4090\n"
|
|
|
|
|
|
def test_a_crlf_traceback_is_not_reduced_to_blank_lines():
|
|
log, _console = _tee(
|
|
['Traceback (most recent call last):\r\n File "run.py", line 3\r\nRuntimeError: boom\r\n']
|
|
)
|
|
assert log.splitlines() == [
|
|
"Traceback (most recent call last):",
|
|
' File "run.py", line 3',
|
|
"RuntimeError: boom",
|
|
]
|
|
|
|
|
|
def test_a_crlf_record_stays_one_json_object():
|
|
log, _console = _tee(['{"event": "model_loaded"}\r\n'])
|
|
assert log == '{"event": "model_loaded"}\n'
|
|
json.loads(log.strip())
|
|
|
|
|
|
def test_a_bar_signing_off_with_a_bare_cr_keeps_its_last_frame():
|
|
# tqdm's close() can leave the terminator on the same write as the final frame.
|
|
log, _console = _tee(["Map: 50%\rMap: 100%\r\n"])
|
|
assert log == "Map: 100%\n"
|
|
|
|
|
|
def test_an_all_blank_line_never_writes_a_carriage_return():
|
|
# The handle appends the platform terminator itself, so a surviving "\r" lands as
|
|
# "\r\r\n" on Windows.
|
|
for chunk in ("\r\n", "\r\r\r\n", " \r \n"):
|
|
log, _console = _tee([chunk])
|
|
assert "\r" not in log, repr(chunk)
|
|
|
|
|
|
def test_a_zero_length_write_does_not_glue_a_frame_onto_the_next_record():
|
|
# print("", end = "") is enough: an empty write used to read as a continuation of the
|
|
# held frame, which then fell through and was written with no newline.
|
|
log, _console = _tee(["\rLoading weights: 47%", "", '{"event": "model_loaded"}\n'])
|
|
lines = log.splitlines()
|
|
assert lines == ["Loading weights: 47%", '{"event": "model_loaded"}']
|
|
json.loads(lines[-1])
|
|
|
|
|
|
def test_the_collapse_matches_the_desktop_reader():
|
|
"""Same rule as collapse_progress_frames in src-tauri/src/process.rs.
|
|
|
|
Settings > Logs offers both sinks side by side, so a line must look the same in either.
|
|
"""
|
|
cases = {
|
|
"plain line": "plain line",
|
|
"a\rb\rc": "c",
|
|
"bar 100%\r": "bar 100%",
|
|
"Map: 50%\rMap: 100%\r ": "Map: 100%",
|
|
"Hardware detected: ROCm": "Hardware detected: ROCm",
|
|
"TAURI_PORT=8888\r": "TAURI_PORT=8888",
|
|
}
|
|
for line, expected in cases.items():
|
|
log, _console = _tee([line + "\n"])
|
|
assert log == expected + "\n", f"{line!r} -> {log!r}, expected {expected!r}"
|