mirror of
https://github.com/unslothai/unsloth.git
synced 2026-08-19 05:44:00 +00:00
* Studio: route every Windows setup line through the UTF-8 stdout sink The desktop setup log rendered "?? Unsloth Studio Setup" over a rule of replacement characters. Tauri spawns Windows PowerShell 5.1 with CREATE_NO_WINDOW (install.rs), so the [Console]::OutputEncoding setter throws and both entry scripts rebind [Console]::Out to a UTF-8 writer. step/substep already write only through that writer when stdout is redirected, so they came out right. Every other line did not: Write-Host is written by 5.1's console host with its own writer on the OEM code page, and U+1F9A5 has no OEM form while U+2500 becomes a bare 0xC4, which from_utf8_lossy turns into U+FFFD. The banner and the footer are not steps, so they kept arriving as mojibake, and install.ps1 had neither the IsOutputRedirected probe nor a mirror at all. Add Write-StudioLine above the first write in studio/setup.ps1 and install.ps1: console handle when redirected, Write-Host when interactive, since it is the only writer that colorizes. Rewrite 164 call sites in setup.ps1 and 155 in install.ps1 onto it, including install.ps1's own step/substep. Write-Host now survives only inside helpers that have already ruled out the redirected sink, and the launcher script install.ps1 generates keeps its own, since it runs as a separate process. No behaviour change for an interactive console user: same text, same colors, same single record per line. test_windows_setup_output_encoding.py gains byte-level coverage that the real banner and footer, sliced out of setup.ps1, survive both launch shapes as valid UTF-8 exactly once, plus a source contract that runs on Linux and names any file:line that reaches for Write-Host outside the allow-list. Studio.Setup.Output.Tests.ps1 covers Write-StudioLine in both modes and pins install.ps1's copy to setup.ps1's. Harnesses that splice these scripts apart now stub or dot-source Write-StudioLine: two PowerShell harnesses, one Python harness, and the VC++ redist leg of studio-windows-inference-smoke. pytest tests/python tests/test_installer_*.py: 1077 passed (2 pre-existing sandbox failures unrelated to this change). All 16 tests/studio harnesses and 57 Pester cases pass. Both scripts parse clean. * CI: spawn install.ps1 as a child process so its lines reach install.log * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Stub the output sink in the llama.cpp backend PowerShell harness * Guard the console-less spawn on a Windows runner The byte-level cases in this file run with a console attached, and a GitHub runner gives a CREATE_NO_WINDOW child one, so the UTF-8 setter succeeds there and every version of these scripts emits a clean banner. Those cases cannot tell this fix from what preceded it. Add cases that call FreeConsole() in the child first, which is the state install.rs's own comment assumes CREATE_NO_WINDOW produces. There Write-Host has no screen buffer to query, throws, and takes the script down: 2 bytes of stdout and exit 1 rather than the banner. The probe is assembled entirely out of text sliced from the script under test and spawned with install.rs's own interpreter, flags and creation flags. No Windows job ran this file, so its byte-level half was only ever exercised under pwsh 7 on the Linux Backend CI leg, which is UTF-8 by default. Add it to the cross-platform parity matrix, which already has a windows-latest row and already triggers on install.ps1 and studio/setup.ps1. * Report skips in the parity step A platform-gated case that stopped running on the row it exists for still reports green with -q alone. * Slice the error preference too It is what turns the Write-Host throw into a dead script rather than a skipped line, so restating it would be assuming the result. * Say what the comments actually mean * Make the console-less cases fail on a lost banner, not just a mangled one * Stub the output sink in every harness that splices these scripts The Write-Host rewrite left four spliced-source harnesses reaching Write-StudioLine without defining it. An undefined command is a terminating error, so each one either aborted or was swallowed by the harness's own catch, and the test kept passing while no longer testing anything. - test_windows_python_venv_hardening.py, partial-rollback case: the five-line split-move warning was lost. The assertion that "both halves are named" only stayed green because $existing is a prefix of the rollback dir, so it matched the dir= line instead. Pin it to the warning text. - test_path_probe_access_denied.ps1, ownership guard: the catch scored the command-not-found as the intended failure and never reached Exit-SetupFailure. Pin the check to the EXIT-SETUP message. - test_windows_installer_concurrency_guard.py: the decision block prints before Exit-InstallFailure, so on Windows the active case aborted at exit 1 and never produced RESULT:blocked. - Studio.Setup.Vs2026.Tests.ps1: on a host without cmake, Ensure-BuildToolsForLlamaSourceBuild hits the sink first and the no-op case fails on the throw. Also stub the three remaining harnesses that splice sink-calling helpers but do not reach the sink on the paths they exercise today, so the next case added to them cannot reintroduce this. --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
330 lines
13 KiB
PowerShell
330 lines
13 KiB
PowerShell
<#
|
|
Pester v5 unit tests for Write-StudioLine / step / substep /
|
|
Write-StudioStdoutMirror in studio/setup.ps1, guarding the desktop setup log
|
|
printing every step twice with the first copy split across two lines:
|
|
|
|
gpu
|
|
none (chat-only / GGUF)
|
|
gpu none (chat-only / GGUF)
|
|
|
|
Two causes. step/substep called Write-Host AND the console mirror, and the
|
|
CLI spawns setup.ps1 as `-Command "& '...' *>&1"`, so both reached the pipe.
|
|
And step's non-VT branch built one line from two Write-Host calls with
|
|
-NoNewline, which a redirected consumer splits at the record boundary.
|
|
|
|
Invariant now: exactly ONE sink. Redirected -> console handle. Interactive
|
|
-> Write-Host. Every other line in both entry scripts goes through
|
|
Write-StudioLine for the same reason: Write-Host is written by 5.1's console
|
|
host on the OEM code page, not by the UTF-8 writer bound to [Console]::Out,
|
|
so the banner and the footer used to arrive as U+FFFD.
|
|
|
|
Pure string formatting, so it runs on any pwsh host. Functions are extracted
|
|
and dot-sourced because setup.ps1 is a top-level installer; a missing one
|
|
FAILS loudly rather than silently passing.
|
|
#>
|
|
|
|
BeforeAll {
|
|
. (Join-Path $PSScriptRoot 'Get-FunctionSource.ps1')
|
|
|
|
$candidates = @(
|
|
$env:SETUP_PS1_PATH,
|
|
(Join-Path $PSScriptRoot '..\..\studio\setup.ps1')
|
|
) | Where-Object { $_ }
|
|
$script:SetupPs1 = $candidates | Where-Object { Test-Path -LiteralPath $_ } | Select-Object -First 1
|
|
if (-not $script:SetupPs1) { throw "Could not locate studio/setup.ps1 (set SETUP_PS1_PATH)." }
|
|
Write-Host "setup.ps1 under test: $script:SetupPs1"
|
|
|
|
$script:InstallPs1 = Join-Path $PSScriptRoot '..\..\install.ps1'
|
|
|
|
foreach ($fn in @('Get-StudioAnsi', 'Write-StudioLine', 'Write-StudioStdoutMirror', 'step', 'substep')) {
|
|
$src = Get-FunctionSource -Path $script:SetupPs1 -Name $fn
|
|
if (-not $src) { throw "Function '$fn' not found in $script:SetupPs1 - cannot test the real code." }
|
|
. ([scriptblock]::Create($src))
|
|
}
|
|
|
|
# Capture the console-handle sink without a real pipe, so the redirected
|
|
# path can be asserted from an ordinary interactive test host.
|
|
function Invoke-CapturingConsoleOut {
|
|
param(
|
|
[Parameter(Mandatory = $true)][scriptblock]$Body,
|
|
[Parameter(Mandatory = $true)][bool]$Redirected
|
|
)
|
|
$script:StudioStdoutRedirected = $Redirected
|
|
$previous = [Console]::Out
|
|
$writer = New-Object System.IO.StringWriter
|
|
try {
|
|
[Console]::SetOut($writer)
|
|
# 6>&1 folds Write-Host into the pipeline so a stray one on the
|
|
# redirected path is caught, not swallowed by the test host.
|
|
$hostRecords = & $Body 6>&1
|
|
} finally {
|
|
[Console]::SetOut($previous)
|
|
}
|
|
[pscustomobject]@{
|
|
Console = $writer.ToString()
|
|
HostRecordCount = @($hostRecords).Count
|
|
# Rendered to strings: 6>&1 yields InformationRecord objects, and the
|
|
# tests care about the text the user would have read.
|
|
HostRecords = @(@($hostRecords) | ForEach-Object { "$_" })
|
|
}
|
|
}
|
|
|
|
# Split on the real line separator only: the split-label bug produced a
|
|
# genuine newline, not a CR redraw.
|
|
#
|
|
# The leading comma is load-bearing. `return @($x)` unrolls a one-element
|
|
# array to a scalar, and a scalar string answers .Count = 1 while [0] gives
|
|
# its first CHARACTER, so a "one line, and it reads X" test would pass the
|
|
# count then compare against a single space.
|
|
function Get-EmittedLines {
|
|
param([string]$Text)
|
|
if ([string]::IsNullOrEmpty($Text)) { return , @() }
|
|
return , @($Text -split "`r?`n" | Where-Object { $_ -ne '' })
|
|
}
|
|
|
|
# Strip comments before asserting a construct is absent: the scripts under
|
|
# test describe -NoNewline in prose, which a naive match would hit.
|
|
function Get-CodeWithoutComments {
|
|
param([string]$Source)
|
|
$stripped = $Source -replace '(?m)#.*$', ''
|
|
return $stripped
|
|
}
|
|
}
|
|
|
|
Describe 'Write-StudioLine is the single sink for every non-step line' {
|
|
BeforeEach {
|
|
$script:StudioVtOk = $false
|
|
$env:NO_COLOR = $null
|
|
}
|
|
|
|
It 'writes to the console handle, and nothing to Write-Host, when redirected' {
|
|
$r = Invoke-CapturingConsoleOut -Redirected $true -Body {
|
|
Write-StudioLine "plain"
|
|
Write-StudioLine "colored" -ForegroundColor Red
|
|
}
|
|
(Get-EmittedLines $r.Console) | Should -Be @('plain', 'colored')
|
|
$r.HostRecordCount | Should -Be 0
|
|
}
|
|
|
|
It 'keeps the banner emoji and the U+2500 rule intact when redirected' {
|
|
$rule = [string]::new([char]0x2500, 52)
|
|
$sloth = [char]::ConvertFromUtf32(0x1F9A5)
|
|
$r = Invoke-CapturingConsoleOut -Redirected $true -Body {
|
|
Write-StudioLine (" " + $sloth + " Unsloth Studio Setup") -ForegroundColor Green
|
|
Write-StudioLine " $rule" -ForegroundColor DarkGray
|
|
}
|
|
$lines = Get-EmittedLines $r.Console
|
|
$lines.Count | Should -Be 2
|
|
$lines[0] | Should -Be " $sloth Unsloth Studio Setup"
|
|
$lines[1] | Should -Be " $rule"
|
|
}
|
|
|
|
It 'emits a blank line as a blank line, not as nothing' {
|
|
$r = Invoke-CapturingConsoleOut -Redirected $true -Body { Write-StudioLine "" }
|
|
$r.Console | Should -Not -BeNullOrEmpty
|
|
$r.Console.Trim() | Should -BeNullOrEmpty
|
|
}
|
|
|
|
It 'never leaks an ANSI escape onto the redirected sink' {
|
|
# Enable-StudioVirtualTerminal returns false without a console handle, so
|
|
# the colored branch must be unreachable there.
|
|
$r = Invoke-CapturingConsoleOut -Redirected $true -Body {
|
|
Write-StudioLine "warning" -ForegroundColor Yellow
|
|
}
|
|
$r.Console | Should -Not -Match ([regex]::Escape([char]27))
|
|
}
|
|
|
|
It 'stays on Write-Host, message intact, when attached to a console' {
|
|
$r = Invoke-CapturingConsoleOut -Redirected $false -Body {
|
|
Write-StudioLine "plain"
|
|
Write-StudioLine "colored" -ForegroundColor Red
|
|
}
|
|
$r.Console | Should -BeNullOrEmpty
|
|
$r.HostRecordCount | Should -Be 2
|
|
$r.HostRecords | Should -Be @('plain', 'colored')
|
|
}
|
|
|
|
It 'still prints an interactive blank line' {
|
|
$r = Invoke-CapturingConsoleOut -Redirected $false -Body { Write-StudioLine "" }
|
|
$r.HostRecordCount | Should -Be 1
|
|
}
|
|
|
|
It 'passes -ForegroundColor through only when the caller supplied one' {
|
|
# An omitted color must not become an empty string: Write-Host cannot
|
|
# bind that to ConsoleColor and the install would abort under "Stop".
|
|
{ Invoke-CapturingConsoleOut -Redirected $false -Body {
|
|
Write-StudioLine "no color here"
|
|
} } | Should -Not -Throw
|
|
}
|
|
}
|
|
|
|
Describe 'step / substep emit exactly one copy when stdout is redirected' {
|
|
BeforeEach {
|
|
$script:StudioVtOk = $false
|
|
$env:NO_COLOR = $null
|
|
}
|
|
|
|
It 'emits a step once, with label and value on the SAME line' {
|
|
$r = Invoke-CapturingConsoleOut -Redirected $true -Body {
|
|
step "gpu" "none (chat-only / GGUF)"
|
|
}
|
|
$lines = Get-EmittedLines $r.Console
|
|
$lines.Count | Should -Be 1
|
|
$lines[0] | Should -Be " gpu none (chat-only / GGUF)"
|
|
}
|
|
|
|
It 'emits a substep once' {
|
|
$r = Invoke-CapturingConsoleOut -Redirected $true -Body {
|
|
substep "installing OXC validator runtime..."
|
|
}
|
|
(Get-EmittedLines $r.Console).Count | Should -Be 1
|
|
}
|
|
|
|
It 'writes NOTHING through Write-Host when redirected (the second copy)' {
|
|
$r = Invoke-CapturingConsoleOut -Redirected $true -Body {
|
|
step "long paths" "enabled"
|
|
substep "detail"
|
|
}
|
|
$r.HostRecordCount | Should -Be 0
|
|
}
|
|
|
|
It 'keeps one line per step across a realistic run' {
|
|
$r = Invoke-CapturingConsoleOut -Redirected $true -Body {
|
|
step "gpu" "none (chat-only / GGUF)"
|
|
step "long paths" "enabled"
|
|
step "git" "git version 2.53.0.windows.2"
|
|
substep "setting up Python environment..."
|
|
}
|
|
(Get-EmittedLines $r.Console).Count | Should -Be 4
|
|
}
|
|
|
|
It 'truncates an over-long label to the 15-column field without wrapping' {
|
|
$r = Invoke-CapturingConsoleOut -Redirected $true -Body {
|
|
step "an-extremely-long-label" "value"
|
|
}
|
|
$lines = Get-EmittedLines $r.Console
|
|
$lines.Count | Should -Be 1
|
|
$lines[0] | Should -Be " an-extremely-lovalue"
|
|
}
|
|
}
|
|
|
|
Describe 'step / substep stay on Write-Host when attached to a console' {
|
|
BeforeEach {
|
|
$script:StudioVtOk = $false
|
|
$env:NO_COLOR = $null
|
|
}
|
|
|
|
It 'writes nothing to the console handle when NOT redirected' {
|
|
$r = Invoke-CapturingConsoleOut -Redirected $false -Body {
|
|
step "gpu" "none (chat-only / GGUF)"
|
|
}
|
|
$r.Console | Should -BeNullOrEmpty
|
|
}
|
|
|
|
It 'emits a step as a SINGLE Write-Host record (not -NoNewline label + value)' {
|
|
$r = Invoke-CapturingConsoleOut -Redirected $false -Body {
|
|
step "gpu" "none (chat-only / GGUF)"
|
|
}
|
|
$r.HostRecordCount | Should -Be 1
|
|
}
|
|
|
|
It 'emits a substep as a single Write-Host record' {
|
|
$r = Invoke-CapturingConsoleOut -Redirected $false -Body {
|
|
substep "detail"
|
|
}
|
|
$r.HostRecordCount | Should -Be 1
|
|
}
|
|
|
|
It 'still emits one record on the ANSI path' {
|
|
$script:StudioVtOk = $true
|
|
$r = Invoke-CapturingConsoleOut -Redirected $false -Body {
|
|
step "gpu" "ok"
|
|
substep "detail"
|
|
}
|
|
$r.HostRecordCount | Should -Be 2
|
|
}
|
|
|
|
It 'still emits one record with NO_COLOR set' {
|
|
$script:StudioVtOk = $true
|
|
$env:NO_COLOR = '1'
|
|
try {
|
|
$r = Invoke-CapturingConsoleOut -Redirected $false -Body {
|
|
step "gpu" "ok"
|
|
}
|
|
$r.HostRecordCount | Should -Be 1
|
|
} finally {
|
|
$env:NO_COLOR = $null
|
|
}
|
|
}
|
|
}
|
|
|
|
Describe 'Write-StudioStdoutMirror honors the resolved sink' {
|
|
It 'writes when the sink says redirected' {
|
|
$r = Invoke-CapturingConsoleOut -Redirected $true -Body {
|
|
Write-StudioStdoutMirror "hello"
|
|
}
|
|
(Get-EmittedLines $r.Console) | Should -Be @('hello')
|
|
}
|
|
|
|
It 'stays silent when the sink says interactive' {
|
|
$r = Invoke-CapturingConsoleOut -Redirected $false -Body {
|
|
Write-StudioStdoutMirror "hello"
|
|
}
|
|
$r.Console | Should -BeNullOrEmpty
|
|
}
|
|
}
|
|
|
|
Describe 'Source contracts that keep the fix from regressing' {
|
|
It 'resolves the redirected sink once, into a script-scoped variable' {
|
|
$source = Get-Content -Raw -LiteralPath $script:SetupPs1
|
|
$source | Should -Match '\$script:StudioStdoutRedirected\s*=\s*\[Console\]::IsOutputRedirected'
|
|
}
|
|
|
|
It 'no longer calls IsOutputRedirected from inside the mirror' {
|
|
# Reading it per-call let the sink disagree with the branch
|
|
# step/substep took; it must be resolved once, up front.
|
|
$src = Get-FunctionSource -Path $script:SetupPs1 -Name 'Write-StudioStdoutMirror'
|
|
$src | Should -Not -Match 'IsOutputRedirected'
|
|
}
|
|
|
|
It 'does not use -NoNewline in setup.ps1 step (record boundaries become newlines)' {
|
|
$src = Get-FunctionSource -Path $script:SetupPs1 -Name 'step'
|
|
(Get-CodeWithoutComments $src) | Should -Not -Match '-NoNewline'
|
|
}
|
|
|
|
It 'does not use -NoNewline in install.ps1 step either' {
|
|
$src = Get-FunctionSource -Path $script:InstallPs1 -Name 'step'
|
|
$src | Should -Not -BeNullOrEmpty
|
|
(Get-CodeWithoutComments $src) | Should -Not -Match '-NoNewline'
|
|
}
|
|
|
|
It 'defines Write-StudioLine in install.ps1 too, with the same body' {
|
|
# install.ps1 cannot dot-source setup.ps1, so it holds a copy. A copy
|
|
# that drifts is a copy that stops routing the installer's own banner.
|
|
$setup = ((Get-FunctionSource -Path $script:SetupPs1 -Name 'Write-StudioLine') -replace '\s+', ' ').Trim()
|
|
$install = ((Get-FunctionSource -Path $script:InstallPs1 -Name 'Write-StudioLine') -replace '\s+', ' ').Trim()
|
|
$setup | Should -Not -BeNullOrEmpty
|
|
$install | Should -Be $setup
|
|
}
|
|
|
|
It 'defines Write-StudioLine before the first line either script prints' {
|
|
# PowerShell resolves functions at call time, but a top-level call above
|
|
# the definition still fails. Comments are stripped first: both scripts
|
|
# name the helper in the prose above it.
|
|
foreach ($path in @($script:SetupPs1, $script:InstallPs1)) {
|
|
$source = Get-CodeWithoutComments (Get-Content -Raw -LiteralPath $path)
|
|
$definition = $source.IndexOf('function Write-StudioLine')
|
|
$definition | Should -BeGreaterThan -1
|
|
$source.IndexOf('Write-StudioLine') | Should -Be ($definition + 'function '.Length)
|
|
}
|
|
}
|
|
|
|
It 'sets the UTF-8 console encoding in both entry scripts' {
|
|
foreach ($path in @($script:SetupPs1, $script:InstallPs1)) {
|
|
$source = Get-Content -Raw -LiteralPath $path
|
|
$source | Should -Match '\[Console\]::OutputEncoding\s*=\s*\$_UnslothUtf8NoBom'
|
|
$source | Should -Match "PYTHONUTF8"
|
|
$source | Should -Match "PYTHONIOENCODING"
|
|
}
|
|
}
|
|
}
|