Commit graph

2 commits

Author SHA1 Message Date
Daniel Han
3ce187da02
Formatting: ruff line-length 100, kwarg-spacing passes, drop blank after short local imports (#6079)
Raise ruff line-length to 100 and extend the local pre-commit format pipeline (def-signature magic-comma normalization, short multi-line assert collapse, kwarg '=' spacing, blank-line-after-short-import removal, adjacent string-literal / f-string+plain merge, redundant-pass pruning). Every transform re-checks the file AST and is dropped if it would differ; the whole-repo reformat is verified AST-identical per file and idempotent.
2026-06-08 04:24:13 -07:00
Wasim Yousef Said
208862218d
feat(studio): training history persistence and past runs viewer (#4501)
* feat(db): add SQLite storage layer for training history

* feat(api): add training history endpoints and response models

* feat(training): integrate DB persistence into training event loop

* feat(ui): add training history views and card grid

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix(studio): address review issues in training history persistence

- Strip hf_token/wandb_token from config before SQLite storage
- Add UUID suffix to job_id for collision resistance
- Use isfinite() for 0.0 metric handling throughout
- Respect _should_stop in error event finalization
- Run schema DDL once per process, not per connection
- Close connection on schema init failure
- Guard cleanup_orphaned_runs at startup
- Cap _metric_buffer at 500 entries
- Make FLUSH_THRESHOLD a class constant
- Map 'running' to 'training' phase in historical view
- Derive LR/GradNorm from history arrays in historical view
- Fix nested button with div[role=button] in history cards
- Guard String(value) against null/undefined in config popover
- Clear selectedHistoryRunId on auto tab switch

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix(studio): address round-2 review findings across training backend and frontend

Backend (training.py):
- Move state mutation after proc.start() so a failed spawn does not wedge
  the backend with is_training=True
- Create DB run row eagerly after proc.start() so runs appear in history
  during model loading, not after first metric event
- Rewrite _flush_metrics_to_db() with snapshot-before-insert pattern to
  preserve metrics arriving during the write and retain buffer on failure
- Guard eval_loss with float() coercion and math.isfinite(), matching the
  existing grad_norm guard
- Increase pump thread join timeout from 3s to 8s to cover SQLite's
  default 5s lock timeout

Frontend (studio-page.tsx):
- Fix history navigation: check isTrainingRunning instead of
  showTrainingView in onSelectRun so completed runs are not misrouted
- Replace activeTab state + auto-switch useEffect with derived tab to
  eliminate react-hooks/set-state-in-effect lint violation

Frontend (historical-training-view.tsx):
- Add explicit "running" branch to message ternary so running runs no
  longer fall through to "Training errored"
- Derive loading from detail/error state and move cleanup to effect
  return to eliminate react-hooks/set-state-in-effect lint violation

Frontend (progress-section.tsx):
- Derive stopRequested from isTrainingRunning && stopRequestedLocal to
  eliminate react-hooks/set-state-in-effect lint violation and remove
  unused useEffect import

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix(studio): resolve 3 remaining bugs from round-2 review

1. Stuck on Current Run tab [12/20]: Only force "current-run" tab when
   isTrainingRunning is true, not when stale completed-run data exists.
   After training ends, users can freely navigate to Configure.

2. Incomplete metric sanitization [7/20]: Apply float() coercion and
   isfinite() guards to loss and learning_rate, matching the existing
   pattern used by grad_norm and eval_loss. Prevents TypeError from
   string values and NaN leaks into history arrays.

3. Stop button state leak across runs [10/20]: Add key={runtime.jobId}
   to ProgressSection so React remounts it when a new run starts,
   resetting stopRequestedLocal state.

* fix(studio): deduplicate loss/lr sanitization in training event handler

Reuse _safe_loss/_safe_lr from the progress update block instead of
re-sanitizing the same raw event values for metric history.

* fix(studio): restore loss > 0 guard to prevent eval steps injecting 0.0 into metric histories

Round-2/3 fixes relaxed the history append guard from `loss > 0` to
`loss is not None`, which let eval-only log events (where loss defaults
to 0.0) append fake zeros into loss_history and lr_history. Restore the
`loss > 0` check to match the worker's own has_train_loss gate. The
float() coercion and isfinite() sanitization from round-3 remain intact.

* fix(studio): resolve training history bugs — nullable loss/lr, tab nav, sparkline

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
2026-03-25 00:58:55 -07:00