unsloth/studio/backend/tests/test_tee_progress_frames.py
Leo Borcherding 7a48774a1d
fix(logs): quiet the polls and drop the duplicated access lines (#8763)
* Studio: quiet the liveness poll and keep access lines out of tauri.log

Measured on an idle 4h desktop session, tauri.log was 5308 lines / 1.14 MB,
of which 4056 (76%) were request_completed access records and 3355 (63%)
came from six polling endpoints. At ~216 bytes a line the 5 MiB cap fills
in about 18 hours, so a failure from the day before had already rotated
out by the time anyone asked for it.

/api/liveness is the desktop watchdog's own probe, roughly every 19s, and
was in no suppressor at all: 760 lines, 14% of the file, saying the
process is still up. Its sibling /api/health was already quiet. Put it in
_QUIET_POLL_PATHS so it heartbeats; non-2xx still logs every time, which
is the case that matters for a watchdog.

Every backend stdout line is written to disk three times: the backend's
own session log, the diagnostics phase log (stream-tagged and stamped),
and tauri.log. The access records carry nothing in the third copy that
the first two lack, so log them at debug!, below the file target's INFO
filter. Non-JSON output -- the startup banner, "Hardware detected:",
stderr, tracebacks -- keeps INFO, which is what anyone opening tauri.log
is there for. The phase log still gets the unfiltered stream, so the
support report loses nothing.

Also:

- Collapse carriage-return progress frames. Reading to \n means a tqdm bar
  arrives as every frame it drew concatenated onto one line; a single
  "Loading weights" bar measured 5086 bytes across three sinks. Keep the
  final frame, which is the one showing 100% and the elapsed total. Cap
  what reaches tauri.log at 16 KiB, the limit the phase log already used.

- Bound logs/llama-server and logs/diffusion-server. Both directories are
  created per model load and were never pruned (319 files going back two
  months here). The server session log has capped itself since it was
  added, so lift that into utils/log_retention.prune_log_dir and call it
  from all three rather than write the loop a third time.

- Log the top-models catalog at debug. Two lists of 40 repo names, one
  line each, on every boot to say a fetch succeeded; the counts say that.

- Configure structlog in run.py before the first line it logs. main.py
  configures on import, several seconds later, so "run_server startup
  begin" rendered through the defaults (console, local time) and
  everything after it as JSON in UTC -- the clock appeared to jump hours
  between the first and second line of one stream.

Not included: the same carriage-return collapse on the Python tee. Doing
it there means line-buffering the file copy, which would withhold a
partial line from server-*.log exactly when the server hangs mid-write.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Collapse progress frames in the session log too, and quiet remote-access

The first pass collapsed carriage-return redraws on the Rust side only. It
left the Python tee alone because doing it there seemed to mean buffering
the file copy until a newline, which would withhold a partial line from
server-*.log exactly when the server hangs mid-write.

It does not have to. Hold back a frame, never a line: an unterminated chunk
containing "\r" is a redraw and is kept until the next one supersedes it,
while an unterminated chunk without one -- a prompt, or a traceback torn by
a hang -- is written the moment it arrives. So the session log gets one
line per bar instead of one per frame, and nothing diagnostic waits on a
newline that may never come.

The console side is untouched: it still sees every frame, which is the
whole point of a progress bar.

Also adds /api/settings/remote-access to the quiet polls (116 lines in the
same 4h sample), for the same reason as the others: it is a plain read of a
toggle, re-read whenever the settings dialog is open.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Install the tee before structlog, and stop three suppressors overreaching

Four review findings, all real.

structlog was configured at import, and PrintLoggerFactory snapshots sys.stdout
while cache_logger_on_first_use freezes that snapshot into any logger that has
already logged. run_server's own logger logs at line one and the tee is not
installed until 45 lines later, so every subsequent run.py line -- the orphan
sweep warnings, the whole rest of startup -- was pinned to the console and never
reached the session log. Configure inside run_server instead, after the tee:
the ordering bug the change set out to fix (line one rendering through
structlog's defaults while line two rendered as JSON in UTC) is still fixed,
and the file gets everything from line one. Moving the tee up also means the
orphan sweep now lands on disk, where it was console-only before.

A withheld progress frame was concatenated onto whatever came next, so a
structlog record arriving mid-redraw landed as Loading 47%{"event": ...} and
stopped being parseable. Close the frame off on its own line unless the next
write is its own redraw or its terminator, and land a frame still held at
close() rather than dropping it.

The desktop watchdog polls every 15s (HEALTH_WATCHDOG_INTERVAL) plus up to a
10s probe budget, so _QUIET_POLL_DEDUP_MS at 10s could never close over two
probes and putting /api/liveness in _QUIET_POLL_PATHS collapsed nothing. Give
it its own 60s window, zeroed alongside the quiet window so --verbose still
logs every probe, and pin the floor with a test that reads both constants out
of commands.rs.

is_backend_access_log_line matched on the event name alone, so it also dropped
non-2xx records to debug! -- below tauri.log's INFO filter. Those are the ones
the backend's heartbeat suppressor deliberately exempts, and a watchdog going
red is what tauri.log is read for. Match only 2xx; a record with no status_code
keeps its INFO line too.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* run.py: keep CRLF lines, stop leaking the tee, import LogConfig lazily

Three defects in the same file, all found while checking this branch against
main. Tests for each alongside.

1. The tee destroyed every CRLF-terminated line.

_last_frame took rpartition("\r")[2] unconditionally, so the "\r" of a CRLF read
as a redraw boundary whose final frame was empty, and the payload went with it:

    "Hardware detected: NVIDIA GeForce RTX 4090\r\n"          -> "\n"
    "Traceback...\r\n  File \"run.py\"...\r\nRuntimeError\r\n" -> "\n\n\n"

On Windows that is every line a child process relays, so a relayed traceback
landed in the session log as blank lines. It is also exactly the artifact #8690
added a UI for. The desktop reader already gets this right: trim_line_endings
strips every trailing \r and \n before collapse_progress_frames sees the text.
Mirror it here so the two sinks stay interchangeable for a reader, and fall back
to the last non-blank frame the same way, so a bar that signs off with a bare \r
keeps its 100% frame instead of writing a blank line.

Also: a "\r" must never reach the file. The handle appends the platform
terminator itself, so one that survives lands as "\r\r\n" on Windows.

2. A zero-length write reopened the record corruption fixed on 08-18.

`data[:1] not in ("", "\r", "\n")` read write("") as a continuation of the held
frame, so the close-off was skipped, the frame fell through to the unterminated
branch, and the next record was glued onto it:

    ["\rLoading weights:  47%", "", '{"event": "model_loaded"}\n']
    -> 'Loading weights:  47%{"event": "model_loaded"}'

print("", end = "") is enough to reach it. Return early on empty data instead.

3. A rejected flag combination left the streams swapped.

--secure --no-cloudflare is a deterministic preflight failure, but it now runs
after _setup_server_disk_logging() has replaced sys.stdout and sys.stderr and
opened a log handle, and nothing unwinds either. An embedder that catches the
SystemExit keeps the tee, and its next run_server() call nests a second one, so
every line is written twice. main does not do this. Validate before the tee.

4. `from loggers.config import LogConfig` at module scope broke Repo tests (CPU).

15 failures in tests/studio/install/test_selection_logic.py, all
"No module named 'loggers.config'; 'loggers' is not a package".
load_studio_run_module stands a bare types.ModuleType in for `loggers` before
exec'ing run.py; a bare module has no __path__, so the import fails during
collection and takes every test in the file with it. That fixture's own comment
names the hazard. LogConfig has one use, inside run_server, so import it there
rather than widening a stub three directories away.

* Prune the log directories after opening the new file, and protect it

Two problems with prune_log_dir as it stands.

The cap never held. All three call sites prune before creating the log, so with
keep=20 the directory settles at 21 and every load repeats the prune-to-20,
create-the-21st cycle. Measured with 25 seeded files and 10 loads: 21 with this
ordering, 20 when the prune runs after the open.

And one odd directory entry disabled retention entirely. The mtime sort ran
inside a single try/except, so a dangling symlink matching the glob raised on
stat() and the whole call returned having deleted nothing. Seeded with 30 files
plus one broken link, keep=5 left all 30. That is the unbounded growth this
helper exists to stop, reintroduced by one bad name. Stat failures are now
per-entry.

Only regular files count now, too: a directory that happens to match the glob
was taking one of the kept slots (25 files plus one such directory, keep=5, left
4 real logs).

prune_log_dir grows a `protect` argument, and the call sites move to after the
open. `protect` is never deleted and counts as one of the `keep`, so the total
stays at keep. It is not enough to trust the new file to sort newest: two loads
in the same second, or a clock that steps back, and it does not.

The retry call site benefits most. The comment right above it explains that the
-try{n} suffix exists so a respawn within the same second cannot truncate the
crash log a retry warning just pointed the user at; pruning before the open on
every attempt worked against that.

New tests cover keep values, the protected file counting toward the cap and
surviving even when it sorts oldest, dangling symlinks, symlinks pointing
outside the directory, directories matching the glob, identical mtimes, family
isolation, and a writer appending across a prune.

* process.rs: CRLF and marker tests, and say what debug! means here

collapse_progress_frames is correct on CRLF because trim_line_endings runs
first, but nothing pinned that, and the Python tee that writes the backend's own
session log got it wrong. Add the cases that hold the two implementations
together: a CRLF payload keeps its text, a CRLF-terminated bar still collapses, a
CRLF JSON record stays parseable and still classifies, a TAURI_PORT marker
survives the collapse whether or not it shares its line with a redraw, and an
all-blank line never keeps a carriage return. Plus the 16 KiB truncation landing
on a character boundary for multi-byte text.

The fallback for an all-blank line returned the whole text, which still holds the
\r the caller asked to collapse away. Return the last frame instead.

Two comment fixes. The new test module had been inserted between the
C:\Windows\system32 comment and the managed_cli_working_dir_tests module it
describes; move it back. And record what debug! actually does for a matched
record: setup_logging in main.rs builds every logger at LevelFilter::Info and
there is no RUST_LOG or any other way to raise it, so log::max_level() is Info
and a matched record is dropped outright rather than demoted, including under
backend --verbose, which makes it emit more of them. That is fine, but it is not
obvious from the call site, and the surrounding comment argued the opposite case
for non-2xx. The records are still written verbatim to the diagnostics phase log
(append_phase_line runs before this decision) and to the backend session log,
both of which the support report and Settings > Logs already offer.

Separately, test_watchdog_window_outlasts_the_probe_interval read the module
global, so it failed for anyone with UNSLOTH_STUDIO_ACCESS_LOG_WATCHDOG_DEDUP_MS
exported; read the default instead. Its two regex lookups now report what moved
in commands.rs rather than raising AttributeError on .group(1).

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Tighten the comments this PR adds

Comment-only pass over the 10 files the PR touches. No code change: the AST gate
(comment_tools.py check --strip-docstrings) reports 9/9 files comments-only, the
Rust file has no non-comment line in the diff, and no docstring was dropped.

114 insertions, 142 deletions. Heaviest in run.py and process.rs, where the same
explanation was being given twice, once in a doc comment and again at the call
site. Every load-bearing why is kept: the structlog factory/cache ordering, why a
CRLF terminator is not a redraw, why the watchdog window has to outlast the poll,
why prune runs after the open, and what debug! means for a matched record.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
2026-08-24 22:21:55 -07:00

178 lines
6.3 KiB
Python

# SPDX-License-Identifier: AGPL-3.0-only
# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved. See /studio/LICENSE.AGPL-3.0
"""The session log's copy of carriage-return progress bars.
A terminal overwrites a redraw in place; a file keeps every frame, so one tqdm bar landed as
kilobytes of near-identical text. The tee keeps the last frame only, and withholds nothing
except frames -- anything without a "\\r" is written the moment it arrives, so a hang cannot
swallow a partial traceback or a prompt.
"""
import io
import json
from run import _TeeStream
class _Sink:
def __init__(self):
self.buf = io.StringIO()
def write(self, data):
self.buf.write(data)
return len(data)
def flush(self):
pass
@property
def text(self):
return self.buf.getvalue()
def _tee(chunks):
log, console = _Sink(), _Sink()
stream = _TeeStream(console, log)
for chunk in chunks:
stream.write(chunk)
return log.text, console.text
def test_plain_output_is_unchanged_on_both_sides():
log, console = _tee(["plain line\n", "another\n"])
assert log == "plain line\nanother\n"
assert console == "plain line\nanother\n"
def test_console_always_sees_every_frame():
_log, console = _tee(["\rbar 1%", "\rbar 50%", "\rbar 100%", "\n"])
# The animation is the console's whole point; only the file copy collapses.
assert console == "\rbar 1%\rbar 50%\rbar 100%\n"
def test_progress_bar_collapses_to_its_final_frame():
log, _console = _tee(["\rbar 1%", "\rbar 50%", "\rbar 100%", "\n"])
assert log == "bar 100%\n"
def test_bar_between_real_lines_keeps_both():
log, _console = _tee(["Loading\n", "\ra 10%", "\ra 99%", "\n", "done\n"])
assert log == "Loading\na 99%\ndone\n"
def test_several_bars_in_one_chunk_collapse_per_line():
log, _console = _tee(["a\rb\rc\nd\re\n"])
assert log == "c\ne\n"
def test_unterminated_prompt_after_a_bar_is_not_withheld():
# "Start Unsloth Studio now? [Y/n]: " never gets a newline; it must still reach the file,
# and on its own line rather than glued to the frame that was being held.
log, _console = _tee(["\rbar 40%", "Start Unsloth Studio now? [Y/n]: "])
assert log == "bar 40%\nStart Unsloth Studio now? [Y/n]: "
def test_record_after_a_held_frame_stays_parseable():
# The reason the frame is closed off rather than prefixed: a structlog record arriving
# while a bar is mid-redraw must still be one JSON object on one line.
log, _console = _tee(["\rLoading weights: 47%", '{"event": "model_loaded"}\n'])
lines = log.splitlines()
assert lines == ["Loading weights: 47%", '{"event": "model_loaded"}']
json.loads(lines[-1])
def test_close_lands_a_frame_nothing_came_back_to_supersede():
log, console = _Sink(), _Sink()
stream = _TeeStream(console, log)
stream.write("\rbar 90%")
stream.close()
assert log.text == "bar 90%\n"
def test_hang_mid_bar_keeps_the_real_partial_line():
# The case that decides whether this is safe: a torn line is written, a frame is not.
log, _console = _tee(["Traceback (most recent call last):", "\rbar 5%"])
assert log == "Traceback (most recent call last):"
def test_file_failure_never_reaches_the_console():
class Exploding(_Sink):
def write(self, data):
raise OSError("disk full")
console = _Sink()
stream = _TeeStream(console, Exploding())
stream.write("still printed\n")
assert console.text == "still printed\n"
# ---------------------------------------------------------------------------------------
# A "\r" is only a redraw when something follows it on the same line.
# ---------------------------------------------------------------------------------------
def test_a_crlf_line_keeps_its_payload():
# "\r\n" is one terminator. Reading its "\r" as a redraw keeps the empty text after it
# and drops the line -- and on Windows every relayed child line arrives in this shape,
# so the session log goes blank exactly where the evidence should be.
log, _console = _tee(["Hardware detected: NVIDIA GeForce RTX 4090\r\n"])
assert log == "Hardware detected: NVIDIA GeForce RTX 4090\n"
def test_a_crlf_traceback_is_not_reduced_to_blank_lines():
log, _console = _tee(
['Traceback (most recent call last):\r\n File "run.py", line 3\r\nRuntimeError: boom\r\n']
)
assert log.splitlines() == [
"Traceback (most recent call last):",
' File "run.py", line 3',
"RuntimeError: boom",
]
def test_a_crlf_record_stays_one_json_object():
log, _console = _tee(['{"event": "model_loaded"}\r\n'])
assert log == '{"event": "model_loaded"}\n'
json.loads(log.strip())
def test_a_bar_signing_off_with_a_bare_cr_keeps_its_last_frame():
# tqdm's close() can leave the terminator on the same write as the final frame.
log, _console = _tee(["Map: 50%\rMap: 100%\r\n"])
assert log == "Map: 100%\n"
def test_an_all_blank_line_never_writes_a_carriage_return():
# The handle appends the platform terminator itself, so a surviving "\r" lands as
# "\r\r\n" on Windows.
for chunk in ("\r\n", "\r\r\r\n", " \r \n"):
log, _console = _tee([chunk])
assert "\r" not in log, repr(chunk)
def test_a_zero_length_write_does_not_glue_a_frame_onto_the_next_record():
# print("", end = "") is enough: an empty write used to read as a continuation of the
# held frame, which then fell through and was written with no newline.
log, _console = _tee(["\rLoading weights: 47%", "", '{"event": "model_loaded"}\n'])
lines = log.splitlines()
assert lines == ["Loading weights: 47%", '{"event": "model_loaded"}']
json.loads(lines[-1])
def test_the_collapse_matches_the_desktop_reader():
"""Same rule as collapse_progress_frames in src-tauri/src/process.rs.
Settings > Logs offers both sinks side by side, so a line must look the same in either.
"""
cases = {
"plain line": "plain line",
"a\rb\rc": "c",
"bar 100%\r": "bar 100%",
"Map: 50%\rMap: 100%\r ": "Map: 100%",
"Hardware detected: ROCm": "Hardware detected: ROCm",
"TAURI_PORT=8888\r": "TAURI_PORT=8888",
}
for line, expected in cases.items():
log, _console = _tee([line + "\n"])
assert log == expected + "\n", f"{line!r} -> {log!r}, expected {expected!r}"