mirror of
https://github.com/unslothai/unsloth.git
synced 2026-08-24 16:23:51 +00:00
41175485bd
* Make a hung launcher say where it is stuck Repo tests (CPU) failed on test_a_reentrant_log_inside_the_delete_retries_does_not_abandon_them with "the launcher was still alive 120s after its signal. Launcher said: (nothing logged after READY)". That message is the same one whether the handler never ran, ran and was refused the pipe, or ran and blocked inside a delete, so it does not narrow anything. The test passes locally, repeatedly, which is exactly the case where the failure message has to carry the evidence. So the children run with PYTHONFAULTHANDLER, and a launcher that overstays its budget gets SIGABRT before SIGKILL. faulthandler prints every thread's stack down the same pipe the tail already reads, using raw fd writes rather than Python's io stack, so it still reports when that io stack is itself the reason for the hang. It also ends the process, so the kill below it stays a backstop. Verified against a launcher whose handler loops forever: the tail goes from "(nothing logged after READY)" to the file and line it is sitting on. Costs nothing when the hang does not happen, since nothing is sent unless the wait times out. * Write the fault dump to a file, not down the pipe that may be the hang PYTHONFAULTHANDLER sends the dump to fd 2, and every launcher here has fd 2 redirected onto the stdout pipe. test_the_handler_survives_a_stdout_nobody_is_ draining deliberately fills that pipe and stops draining it, which is one of the hangs most worth diagnosing, and a raw write bypasses Python's io lock but not pipe backpressure. SIGABRT would have killed the child with nothing written, and the failure message would have been as empty as the one this exists to replace: a diagnostic silenced by the very fault it is describing. So faulthandler is armed against a file, one per test beside its other artefacts, by a preamble every generated runner carries, and the assertion prints it. Verified against that exact scenario, a handler that floods stdout forever while nobody reads: the file holds the stack, and the stderr route wrote nothing at all. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> |
||
|---|---|---|
| .. | ||
| studio_gpu | ||
| t4_smoke | ||
| test_launch_cleanup.py | ||
| test_studio_gpu_harness.py | ||
| test_t4_ci_transport.py | ||
| test_t4_payload_assertions.py | ||
| test_t4_smoke_harness.py | ||