mirror of
https://github.com/unslothai/unsloth.git
synced 2026-07-09 15:58:41 +00:00
e27cc0ab08
* studio/ci: npm tarball content scanner (no-install, hostile-input safe) Counterpart to scripts/scan_packages.py for the npm side. Pip-side scanner reads requirements files, downloads PyPI archives via `pip download --no-deps`, and pattern-scans them for malicious shapes. This change adds the equivalent for npm tarballs. Why === PR #5392 (lockfile_supply_chain_audit.py) catches injection-pattern attacks where the malicious metadata lives IN the lockfile -- e.g. the TanStack Shai-Hulud worm that injected an `optionalDependencies` entry pointing at a GitHub commit. It does not catch the broader class of "legit-registry tarball with malicious content but normal lockfile metadata": attacker steals a maintainer's npm publish token, publishes a malicious version to registry.npmjs.org with a valid integrity hash, and the lockfile entry looks normal -- the malicious code lives inside the tarball's dist/index.js or its own postinstall script. Today that gap is covered reactively by `npm audit` + OSV-Scanner once the GHSA lands; there is a real window before that. This scanner closes the window by inspecting tarball CONTENT. What it checks ============== For each entry in studio/frontend/package-lock.json: 1. Download the tarball directly from registry.npmjs.org. Refuse any non-allowlisted URL. Stream-bounded at 64 MiB. 2. Verify SHA-512 integrity against the lockfile entry BEFORE opening the tarball. 3. Safely extract into a sandboxed temp dir behind guards: - reject symlinks / hardlinks (LNKTYPE, SYMTYPE) - reject absolute paths and `..` traversal - reject character / block / FIFO devices - per-file size cap 8 MiB, cumulative cap 128 MiB, member count cap 50000 - stream open (mode='r|gz') so we abort mid-extract - extracted files set to non-executable mode (0o644) 4. Pattern-scan the extracted text content for: - lifecycle (preinstall/install/postinstall/prepare) scripts in any package.json that fetch + pipe-to-shell external content -- the install-time RCE vector - optionalDependencies pointing at github: / git+ / git: (TanStack worm injection shape) - C2 / exfiltration hosts: getsession.org, 169.254.169.254 (IMDS), 169.254.170.2 (ECS), metadata.google.internal, vault.svc.cluster.local, k8s ServiceAccount token paths, ACTIONS_ID_TOKEN_REQUEST_URL/TOKEN, npm publish-token enumeration endpoint - credential paths a frontend lib should never read: ~/.npmrc, ~/.aws/credentials, ~/.ssh/id_*, /.kube/config, /.docker/config.json - JS regex: Function/eval against base64-decoded payload, process.env.GITHUB_TOKEN / NPM_TOKEN / AWS_* access in package source - obfuscation: large base64-ish blob (>=2 KiB) fed into Function or eval (router_init.js dropper shape) - literal IOC substrings from public advisories Safety ====== Threat model: every tarball is hostile. The scanner: - never runs `npm install`, never executes anything from a downloaded tarball, never calls subprocess on extracted content - downloads only from registry.npmjs.org (defence-in-depth check at parse time AND inside download_tarball) - stdlib-only (no third-party deps -- adding one would itself be a supply-chain liability) - tempdir wiped via atexit on every termination path - exit codes: 0 clean, 1 HIGH/CRITICAL finding, 2 internal error Wiring ====== New job `npm-scan-packages` in security-audit.yml, parallel to `pip-scan-packages`. Triggers same as the existing audits (PR on manifest changes, push to main/pip, daily 04:13 UTC, dispatch). Initially `continue-on-error: true` so the baseline can settle -- matches the existing convention for the other audit steps. Drop that flag once the baseline is clean for a week. Verified locally ================ - AST parse OK. - Real-network 3-package smoke: 0 findings. - Real-network 25-package smoke (Babel + assistant-ui surface): 0 findings, no hard errors. - 9 fault-injection scenarios all pass: 1. zip-slip path traversal refused 2. symlink member refused 3. oversized member refused (size cap) 4. too-many-members refused (count cap) 5. router_init.js IOC + obfuscated-blob shape both detected in synthetic malicious tarball 6. lifecycle fetch-exec in scripts.preinstall detected as CRITICAL 7. AWS IMDS reference (169.254.169.254) detected 8. SRI integrity-parser accepts syntactically-valid SRI 9. download_tarball refuses non-allowlisted hostname Refs ==== - https://tanstack.com/blog/npm-supply-chain-compromise-postmortem - https://github.com/TanStack/router/issues/7383 - https://github.com/TanStack/router/security/advisories/GHSA-g7cv-rxg3-hmpx - https://www.aikido.dev/blog/mini-shai-hulud-is-back-tanstack-compromised - https://www.stepsecurity.io/blog/mini-shai-hulud-is-back-a-self-spreading-supply-chain-attack-hits-the-npm-ecosystem * scan_npm_packages: kill false positives + handle real native binaries First CI run on PR #5393 (run 25710423126 / job 75489317395) hit two false-positive classes plus one cap-too-tight class: False positives (7 findings): @langchain/core 1.1.44 ssrf.{cjs,js}: a SSRF *protection* module that ships a literal blocklist `const CLOUD_METADATA_IPS = [...]` of IMDS hosts as data the library REFUSES to dial. Our scanner saw the IPs as substrings and flagged 6 of them. object-treeify 1.1.33 package.json: a manual `docker` dev script that mounts `~/.npmrc` and `~/.aws` for local containerised builds. npm never runs `scripts.docker` automatically; it is only invoked when a developer runs `npm run docker`. Our bare substring scan flagged the `/.npmrc` reference anyway. Cap-too-tight class (10+ findings): next/swc, rolldown bindings, biome CLI, lightningcss, mermaid sourcemap, typescript.js. The 8 MiB per-file cap was calibrated for JS source and rejected legitimate precompiled native binaries (next-swc .node is 137 MB) and CLI executables (biome is 25-33 MB). Fixes ===== cred-surface-host detection split into two tiers: ALWAYS_BAD substrings have no legitimate use anywhere and still bare-match: `registry.npmjs.org/-/npm/v1/tokens`, `ACTIONS_ID_TOKEN_REQUEST_URL/TOKEN`. NEEDS_CONTEXT substrings (IMDS IPs, GCE metadata host, k8s ServiceAccount path, Vault endpoint) require co-occurrence with EITHER a fetch verb (fetch/axios/http.get/etc) within 200 chars OR an `http(s)?://HOST` URL prefix OR a `host:`/`hostname:` config field. A defensive blocklist literal does not match any of those rules; an actual outbound call always does. cred-surface-path detection moved out of the bare-text scan into `scan_package_json` and scoped to the 4 NPM lifecycle hooks (preinstall / install / postinstall / prepare). A `/.npmrc` reference in a `docker` dev script is silent; a `cat ~/.npmrc | curl ...` in a `postinstall` fires HIGH. Per-file size cap split by content type, sniffed via 16-byte magic header read (ELF / Mach-O / PE / WASM / archive formats), plus suffix list (.node/.wasm/.so/.dll/.dylib/.exe), plus regex for versioned shared libs (libfoo.so.8.17.3), plus a null-byte ratio fallback for extensionless binaries that headers do not catch. Text files: 16 MiB cap (still tight; typescript.js at 9.1 MB is the legitimate ceiling). Binary files: 256 MiB cap (next-swc .node is 137 MB; sharp libvips is ~18 MB; rolldown bindings are 18-26 MB each). Cumulative: 512 MiB per tarball. Tarball: 256 MiB compressed. Binary files are also skipped in the content scanner -- regex over compiled machine code is noise. The IOC substring fallback in `scan_extracted_tree` now uses the same magic-sniff to decide whether to grep. HTTP timeout bumped 30s -> 60s for large tarballs. Verified ======== - AST parse OK. - 11 fault-injection tests pass: * zip-slip, symlink, oversized-declared-size, count-cap * router_init.js IOC detected * IMDS-in-URL still detected (new contextual rule) * langchain SSRF blocklist no longer false-positive * object-treeify docker script no longer false-positive * lifecycle-script `cat ~/.npmrc | curl ...` detected * synthetic ELF (extensionless executable) extracts and is correctly skipped from text scan * versioned `.so.8.17.3` shared lib extracts cleanly - Real-network end-to-end on the full lockfile: 968 packages, 0 findings, 0 hard errors, 76 seconds. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> |
||
|---|---|---|
| .. | ||
| data | ||
| enforce_kwargs_spacing.py | ||
| install_gemma4_mlx.sh | ||
| install_qwen3_6_mlx.sh | ||
| lockfile_supply_chain_audit.py | ||
| notebook_to_python.py | ||
| notebook_validator.py | ||
| run_ruff_format.py | ||
| scan_npm_packages.py | ||
| scan_packages.py | ||
| stamp_studio_release.py | ||