mirror of
https://github.com/ruvnet/RuVector.git
synced 2026-07-09 17:28:42 +00:00
feat(sona): metaharness-Darwin evolves EWC++ config beyond hand-tuned SOTA (#615)
* feat(sona): metaharness-Darwin evolves EWC++ config beyond hand-tuned SOTA examples/darwin_ewc: applies the Meta-Harness 'freeze the model, evolve the harness' pattern to SONA's continual-learning layer — frozen = the EWC++ algorithm (EwcPlusPlus), evolved = its EwcConfig genome (lambda schedule, Fisher decay, auto task-boundary threshold, learning rate). Benchmark: a single weight vector trained on a sequence of tasks (no replay, auto-detected boundaries) — the canonical plasticity-vs-forgetting frontier. Darwin (GA + coordinate-descent polish) evolves the genome on TRAIN task- sequences; results reported on HELD-OUT sequences (different seeds). Measured (deterministic), held-out: the evolved config beats EwcConfig::default() (the crate's hand-tuned 'OPTIMIZED' values) by 35% lower final loss and 98.6% less forgetting — a strict Pareto win (plasticity also improves), and it generalizes to unseen task sequences. clippy -D warnings clean, fmt clean. Co-Authored-By: claude-flow <ruv@ruv.net> * feat(sona): weightAdapter gene — Darwin selects/prunes a fine-tuned adapter Extends the metaharness-Darwin line: expose a fine-tuned adapter (e.g. a LoRA distilled from verified SWE-bench trajectories — the 'autonomous data engine') as a gene (which_adapter, alpha) so evolutionary selection decides whether/how much to apply it (w_eff = w_base + alpha·Δw) instead of assuming new weights are better. examples/darwin_weightadapter demonstrates it on two conflicting domains with a generalizing adapter and an overfit one. Key finding (sharpens the idea): 'selection prunes overfit adapters' holds ONLY under per-domain evaluation. Measured (held-out, in-dist-majority eval): overfit α=0.55 → ΔA +0.249 / ΔB -0.357 (regresses out-dist) AGGREGATE (volume-weighted) fitness → picks the overfit adapter (silent B regression) PER-DOMAIN (no-regression Pareto) → prunes it, keeps the generalizing adapter So: evolve the adapter as a gene, but score it per-repository. clippy/fmt clean. Co-Authored-By: claude-flow <ruv@ruv.net> * docs(adr): ADR-271 metaharness-Darwin for SONA self-improvement Documents the metaharness-Darwin-evolves-SONA architecture: EWC++ config evolution (PR #615), the weightAdapter gene (per-domain Pareto selection of fine-tuned adapters), the Autonomous Data Engine (execution-verified SWE-bench trajectories -> DPO pairs), and four Ornith-1.0 borrows (immutable-boundary + deterministic-monitor-with-exclude-from-advantage + frozen-LLM-judge-veto reward-hacking defense; per-task-category specialization; two-stage scaffold reward credit; staleness-weighted replay). Method-not-model: external evolutionary vs Ornith's in-weights RL. Co-Authored-By: claude-flow <ruv@ruv.net> * feat(sona): darwin-guard reward-hacking defense (Ornith-1.0 borrow, ADR-271) 3-layer defense for evolutionary config search: (1) immutable verifier boundary (screen is a pure fn of verifier output the candidate can't fabricate); (2) deterministic monitor — non-finite / out-of-bounds / degenerate candidates are EXCLUDED from selection (best_accepted), not zero-scored, so a hack can neither win nor bias the advantage; (3) IntentJudge trait = frozen-LLM veto-only layer. Wired into darwin_ewc: NaN/collapsed configs are excluded from the GA ranking (also fixes the partial_cmp().unwrap() NaN-panic). 4 unit tests; benchmark still reaches beyond-SOTA (35% lower loss, 98.6% less forgetting) unchanged. clippy -D warnings + fmt clean. Co-Authored-By: claude-flow <ruv@ruv.net> * feat(sona): per-task-category genome router beats single global config (ADR-271) Ornith-1.0 borrow #2 (per-category specialization): evolve a router task-class -> genome instead of one global EwcConfig. Two continual-learning workload classes with conflicting optima (STABLE wants high lambda / retain; VOLATILE wants low lambda / stay plastic). Guard-screened evolution. Measured (held-out, adequate per-class data): per-category router 0.1122 vs single best global genome 0.1144 -> router ~1.9% better on unseen sequences, because one config cannot serve conflicting workloads. Honest caveat (discovered + documented): the gain REVERSES when per-class data is scarce — a specialized config overfits while the pooled global generalizes. Per-category routing needs enough per-category samples (Ornith's regime). ADR-271 updated; clippy/fmt clean. Co-Authored-By: claude-flow <ruv@ruv.net> * feat(sona): online auto-tuner with staleness-weighted replay (ADR-271, Ornith borrow #4) auto_tuner module: StalenessSchedule (Ornith w(d_t): fresh<=k1, exp-decay, drop>k2) + StalenessWindow (staleness-weighted running estimate of recent config performance, evicts stale obs). 4 unit tests. examples/darwin_autotuner: a (1+1)-ES that adapts a DEPLOYED EwcConfig to a drifting workload stream (regime A -> B at the midpoint), scoring the incumbent on the staleness window and accepting a perturbation only when it beats the recent score. Measured: online tuner ~3% lower post-drift loss than the static deployment config (10 accepted re-tunes). Margin is modest on synthetic regimes; the durable win is the reusable staleness machinery + the online-adaptation principle (a fixed offline-tuned config goes stale under drift). Completes the four ADR-271 components. clippy --all-targets -D warnings + fmt clean; 102 sona tests pass. Co-Authored-By: claude-flow <ruv@ruv.net> * feat(sona): contamination/disjointness guard in darwin-guard (weight-eft/ADR-198 borrow) Adds the train/eval contamination guard — the gap @metaharness/weight-eft exposed in our reward-hacking-only guard. contamination()/assert_train_eval_disjoint() fail on any train∩eval instance-ID overlap (training/selecting on eval instances is fake lift); filter_holdout() partitions a set disjoint-by-construction and surfaces what was excluded. The SONA-side analog of weight-eft's assertTrainEvalDisjoint. 2 new tests (6 total in darwin_guard). ADR-271 updated: §3 Data Engine now cites @metaharness/weight-eft + adopts its RLHF-correct recipe (SFT distills ALL gold incl. off-policy frontier successes; DPO ON-POLICY cheap-vs-cheap only), and the darwin-guard borrow gains layer (iv) the contamination disjointness guard. clippy -D warnings + fmt clean. Co-Authored-By: claude-flow <ruv@ruv.net> * chore(release): ruvector-sona 0.2.1 — darwin_guard + auto_tuner modules Non-breaking minor feature release (new public modules darwin_guard, auto_tuner). Patch bump keeps the ^0.2 requirement of all in-workspace dependents (ruvllm, rvlite, mcp-brain, ...) satisfied. Co-Authored-By: claude-flow <ruv@ruv.net> --------- Co-authored-by: ruvnet <ruvnet@gmail.com>
This commit is contained in:
rUv
GitHub
parent
a67f3564e2
commit
b2a32eae2f
9 changed files with 1702 additions and 1 deletions
|
|
@ -1,6 +1,6 @@
|
|||
[package]
|
||||
name = "ruvector-sona"
|
||||
version = "0.2.0"
|
||||
version = "0.2.1"
|
||||
edition = "2021"
|
||||
rust-version = "1.70"
|
||||
authors = ["RuVector Team <team@ruvector.dev>"]
|
||||
|
|
|
|||
251
crates/sona/examples/darwin_autotuner.rs
Normal file
251
crates/sona/examples/darwin_autotuner.rs
Normal file
|
|
@ -0,0 +1,251 @@
|
|||
//! Online auto-tuner for SONA's EWC config (ADR-271, Ornith-1.0 borrow #4).
|
||||
//!
|
||||
//! The point of *online* tuning is **non-stationarity**: a config tuned once,
|
||||
//! offline, goes stale when the workload drifts. Here a `(1+1)`-ES re-tunes the
|
||||
//! `EwcConfig` against a LIVE, drifting trajectory stream using the
|
||||
//! staleness-weighted window `w(d_t)` from `ruvector_sona::auto_tuner`: it scores
|
||||
//! the incumbent on *recent* observations, and accepts a perturbation only when
|
||||
//! it beats that recent score — so it tracks a moving optimum instead of
|
||||
//! averaging over a stale past.
|
||||
//!
|
||||
//! Drift scenario: the stream runs in regime A (small task shifts → wants high
|
||||
//! lambda / retain) for the first half, then drifts to regime B (large shifts →
|
||||
//! wants low lambda / stay plastic). We compare cumulative POST-DRIFT loss of:
|
||||
//! * the best STATIC config (tuned offline for the deployment regime A), vs
|
||||
//! * the ONLINE auto-tuner (adapts as the regime drifts).
|
||||
//! Beyond-static result: the online tuner's post-drift loss is lower — adaptation
|
||||
//! beats any fixed config under drift.
|
||||
//!
|
||||
//! Run: `cargo run -p ruvector-sona --release --example darwin_autotuner`
|
||||
|
||||
use rand::rngs::StdRng;
|
||||
use rand::{Rng, SeedableRng};
|
||||
use ruvector_sona::auto_tuner::{StalenessSchedule, StalenessWindow};
|
||||
use ruvector_sona::{EwcConfig, EwcPlusPlus};
|
||||
|
||||
const PARAM_COUNT: usize = 96;
|
||||
const EPOCHS: usize = 80;
|
||||
const RETUNE_EVERY: usize = 2;
|
||||
|
||||
#[derive(Clone, Copy)]
|
||||
struct Regime {
|
||||
n_tasks: usize,
|
||||
steps: usize,
|
||||
shift: f32,
|
||||
}
|
||||
const REGIME_A: Regime = Regime {
|
||||
n_tasks: 3,
|
||||
steps: 90,
|
||||
shift: 0.25,
|
||||
}; // stable
|
||||
const REGIME_B: Regime = Regime {
|
||||
n_tasks: 8,
|
||||
steps: 30,
|
||||
shift: 1.4,
|
||||
}; // volatile
|
||||
|
||||
fn regime_at(epoch: usize) -> Regime {
|
||||
if epoch < EPOCHS / 2 {
|
||||
REGIME_A
|
||||
} else {
|
||||
REGIME_B
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Clone)]
|
||||
struct Genome {
|
||||
lr: f32,
|
||||
initial_lambda: f32,
|
||||
fisher_ema_decay: f32,
|
||||
boundary_threshold: f32,
|
||||
}
|
||||
impl Genome {
|
||||
fn baseline() -> Self {
|
||||
let d = EwcConfig::default();
|
||||
Self {
|
||||
lr: 0.08,
|
||||
initial_lambda: d.initial_lambda,
|
||||
fisher_ema_decay: d.fisher_ema_decay,
|
||||
boundary_threshold: d.boundary_threshold,
|
||||
}
|
||||
}
|
||||
fn to_config(&self) -> EwcConfig {
|
||||
EwcConfig {
|
||||
param_count: PARAM_COUNT,
|
||||
max_tasks: 10,
|
||||
initial_lambda: self.initial_lambda,
|
||||
min_lambda: 50.0,
|
||||
max_lambda: 30_000.0,
|
||||
fisher_ema_decay: self.fisher_ema_decay,
|
||||
boundary_threshold: self.boundary_threshold,
|
||||
gradient_history_size: 60,
|
||||
}
|
||||
}
|
||||
fn mutate(&self, rng: &mut StdRng) -> Self {
|
||||
let mut c = self.clone();
|
||||
if rng.gen::<f32>() < 0.6 {
|
||||
c.lr = (c.lr * (0.6 + rng.gen::<f32>() * 0.8)).clamp(0.01, 0.5);
|
||||
}
|
||||
if rng.gen::<f32>() < 0.7 {
|
||||
c.initial_lambda =
|
||||
(c.initial_lambda * (0.3 + rng.gen::<f32>() * 1.6)).clamp(10.0, 20_000.0);
|
||||
}
|
||||
if rng.gen::<f32>() < 0.4 {
|
||||
c.fisher_ema_decay =
|
||||
(c.fisher_ema_decay + rng.gen::<f32>() * 0.04 - 0.02).clamp(0.9, 0.9999);
|
||||
}
|
||||
if rng.gen::<f32>() < 0.5 {
|
||||
c.boundary_threshold =
|
||||
(c.boundary_threshold + rng.gen::<f32>() * 2.0 - 1.0).clamp(0.5, 6.0);
|
||||
}
|
||||
c
|
||||
}
|
||||
}
|
||||
|
||||
/// One epoch = a continual-learning sequence under a regime; returns avg final loss.
|
||||
fn run_epoch(regime: &Regime, g: &Genome, seed: u64) -> f32 {
|
||||
let p = PARAM_COUNT;
|
||||
let mut rng = StdRng::seed_from_u64(seed);
|
||||
let targets: Vec<Vec<f32>> = (0..regime.n_tasks)
|
||||
.map(|_| {
|
||||
(0..p)
|
||||
.map(|_| regime.shift * rng.gen_range(-1.0f32..1.0))
|
||||
.collect()
|
||||
})
|
||||
.collect();
|
||||
let mut ewc = EwcPlusPlus::new(g.to_config());
|
||||
let mut w = vec![0.0f32; p];
|
||||
for target in &targets {
|
||||
for _ in 0..regime.steps {
|
||||
let grad: Vec<f32> = w.iter().zip(target).map(|(wi, ti)| wi - ti).collect();
|
||||
if ewc.detect_task_boundary(&grad) {
|
||||
ewc.start_new_task();
|
||||
}
|
||||
let gc = ewc.apply_constraints(&grad);
|
||||
for (wi, gi) in w.iter_mut().zip(gc.iter()) {
|
||||
*wi -= g.lr * gi;
|
||||
}
|
||||
ewc.update_fisher(&grad);
|
||||
ewc.set_optimal_weights(&w);
|
||||
}
|
||||
}
|
||||
targets
|
||||
.iter()
|
||||
.map(|tg| {
|
||||
0.5 * w
|
||||
.iter()
|
||||
.zip(tg)
|
||||
.map(|(a, b)| (a - b) * (a - b))
|
||||
.sum::<f32>()
|
||||
/ p as f32
|
||||
})
|
||||
.sum::<f32>()
|
||||
/ regime.n_tasks as f32
|
||||
}
|
||||
|
||||
/// Offline-tune the best STATIC config for the DEPLOYMENT regime (A) — "tuned
|
||||
/// once before deployment, then the world drifts to B." This is the config that
|
||||
/// goes stale; the online tuner must adapt past it.
|
||||
fn best_static() -> Genome {
|
||||
let mut rng = StdRng::seed_from_u64(0x57A71C);
|
||||
let seeds: Vec<u64> = (0..12).collect();
|
||||
let score = |g: &Genome| {
|
||||
seeds
|
||||
.iter()
|
||||
.map(|&s| run_epoch(®IME_A, g, s))
|
||||
.sum::<f32>()
|
||||
/ seeds.len() as f32
|
||||
};
|
||||
let mut best = Genome::baseline();
|
||||
let mut best_s = score(&best);
|
||||
for _ in 0..200 {
|
||||
let cand = best.mutate(&mut rng);
|
||||
let s = score(&cand);
|
||||
if s < best_s {
|
||||
best = cand;
|
||||
best_s = s;
|
||||
}
|
||||
}
|
||||
best
|
||||
}
|
||||
|
||||
fn main() {
|
||||
println!("== SONA · online auto-tuner (staleness-weighted (1+1)-ES, Ornith w(d_t)) ==");
|
||||
println!(
|
||||
"drift: regime A (stable) for epochs 0..{}, regime B (volatile) after\n",
|
||||
EPOCHS / 2
|
||||
);
|
||||
|
||||
let static_cfg = best_static();
|
||||
println!(
|
||||
"best STATIC config (offline, deployment regime A): λ0={:.0} lr={:.3} bθ={:.2}",
|
||||
static_cfg.initial_lambda, static_cfg.lr, static_cfg.boundary_threshold
|
||||
);
|
||||
|
||||
// Online auto-tuner: (1+1)-ES over the live stream, scored on a staleness window.
|
||||
let mut rng = StdRng::seed_from_u64(0xA070);
|
||||
// Deploy the offline-tuned config, then let the tuner adapt it online.
|
||||
let mut current = static_cfg.clone();
|
||||
let mut window = StalenessWindow::new(StalenessSchedule::new(6, 40, 0.10), 64);
|
||||
|
||||
let (mut post_static, mut post_online) = (0.0f32, 0.0f32);
|
||||
let mut accepts = 0;
|
||||
for epoch in 0..EPOCHS {
|
||||
let regime = regime_at(epoch);
|
||||
let seed = 7000 + epoch as u64;
|
||||
|
||||
// Incumbent runs the live epoch; record into the staleness window.
|
||||
let loss_online = run_epoch(®ime, ¤t, seed);
|
||||
window.push(loss_online);
|
||||
let loss_static = run_epoch(®ime, &static_cfg, seed);
|
||||
|
||||
// Accumulate POST-DRIFT loss (the regime the static config wasn't tuned for).
|
||||
if epoch >= EPOCHS / 2 {
|
||||
post_online += loss_online;
|
||||
post_static += loss_static;
|
||||
}
|
||||
|
||||
// (1+1)-ES re-tune: probe a perturbation on the *recent* regime; accept if
|
||||
// it beats the incumbent's staleness-weighted recent score.
|
||||
if epoch % RETUNE_EVERY == 0 && epoch > 0 {
|
||||
if let Some(incumbent_recent) = window.weighted_mean() {
|
||||
let cand = current.mutate(&mut rng);
|
||||
// Probe the candidate on a few recent-regime sequences (online: no peeking ahead).
|
||||
let probe: f32 = (0..3)
|
||||
.map(|k| run_epoch(®ime, &cand, seed.wrapping_add(31 * k + 1)))
|
||||
.sum::<f32>()
|
||||
/ 3.0;
|
||||
if probe < incumbent_recent {
|
||||
current = cand;
|
||||
window.clear_samples(); // score the new config on its own fresh samples
|
||||
accepts += 1;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
let half = (EPOCHS / 2) as f32;
|
||||
println!(
|
||||
"online tuner: λ0={:.0} lr={:.3} bθ={:.2} ({accepts} accepted re-tunes)",
|
||||
current.initial_lambda, current.lr, current.boundary_threshold
|
||||
);
|
||||
println!("\n-- POST-DRIFT cumulative loss (regime B; mean/epoch) --");
|
||||
println!(" static config (offline-tuned): {:.4}", post_static / half);
|
||||
println!(" online auto-tuner : {:.4}", post_online / half);
|
||||
let gain = (post_static - post_online) / post_static * 100.0;
|
||||
println!(" → online tuner is {gain:.1}% better after the workload drift");
|
||||
println!(
|
||||
"{}",
|
||||
if post_online < post_static - 1e-4 {
|
||||
"BEYOND STATIC — staleness-weighted online tuning tracks the drift a fixed config cannot"
|
||||
} else {
|
||||
"no improvement over the static config this run"
|
||||
}
|
||||
);
|
||||
println!(
|
||||
" (the margin is modest here — these synthetic regimes lack cleanly-opposite optima;\n \
|
||||
the reusable win is the staleness-weighted `auto_tuner` machinery + the online ES\n \
|
||||
that adapts a deployed config to drift instead of going stale.)"
|
||||
);
|
||||
assert!(post_online.is_finite() && post_static.is_finite());
|
||||
}
|
||||
366
crates/sona/examples/darwin_ewc.rs
Normal file
366
crates/sona/examples/darwin_ewc.rs
Normal file
|
|
@ -0,0 +1,366 @@
|
|||
//! Meta-Harness Darwin applied to SONA's EWC++ — "freeze the model, evolve the
|
||||
//! harness", where the *frozen model* is the EWC++ continual-learning algorithm
|
||||
//! ([`EwcPlusPlus`]) and the *evolved harness* is its [`EwcConfig`] genome
|
||||
//! (lambda schedule, Fisher decay, auto-boundary threshold, learning rate).
|
||||
//!
|
||||
//! ## The benchmark (a real plasticity/forgetting frontier)
|
||||
//!
|
||||
//! A single weight vector `w` is trained on a *sequence* of tasks, each with its
|
||||
//! own random target. The learner only ever sees the *current* task's gradient
|
||||
//! (no replay) and must detect task switches itself (auto boundary detection via
|
||||
//! `boundary_threshold`). EWC++ projects gradients away from parameters its
|
||||
//! online Fisher deems important to earlier tasks. This is the canonical
|
||||
//! continual-learning setup:
|
||||
//!
|
||||
//! * low lambda → `w` chases the latest task → high plasticity, **forgets**;
|
||||
//! * high lambda → `w` frozen near task 0 → **can't learn** later tasks.
|
||||
//!
|
||||
//! The score is the average final loss across ALL tasks under one `w` (plus a
|
||||
//! forgetting penalty) — minimised only by a config that *both* learns and
|
||||
//! retains.
|
||||
//!
|
||||
//! ## Beyond SOTA (precise claim)
|
||||
//!
|
||||
//! `EwcConfig::default()` ships hand-tuned "OPTIMIZED" values (lambda 2000, …).
|
||||
//! Darwin evolves the genome on TRAIN task-sequences and we report the result on
|
||||
//! HELD-OUT sequences (different seeds). The beyond-SOTA result is: the evolved
|
||||
//! genome beats the hand-tuned default on *unseen* task sequences — i.e. the
|
||||
//! metaharness loop out-tunes the crate's hand-tuning, and it generalises.
|
||||
//!
|
||||
//! Run: `cargo run -p ruvector-sona --release --example darwin_ewc`
|
||||
|
||||
use rand::rngs::StdRng;
|
||||
use rand::{Rng, SeedableRng};
|
||||
use ruvector_sona::darwin_guard::{Guard, Verdict};
|
||||
use ruvector_sona::{EwcConfig, EwcPlusPlus};
|
||||
|
||||
const PARAM_COUNT: usize = 128;
|
||||
const N_TASKS: usize = 6;
|
||||
const STEPS_PER_TASK: usize = 80;
|
||||
const TRAIN_SEEDS: &[u64] = &[1, 2, 3, 4, 5, 6, 7, 8];
|
||||
const TEST_SEEDS: &[u64] = &[101, 102, 103, 104, 105, 106]; // held out
|
||||
|
||||
// ── Genome: the EwcConfig harness Darwin evolves (frozen = the EWC++ algorithm) ─
|
||||
#[derive(Clone, Debug)]
|
||||
struct Genome {
|
||||
lr: f32,
|
||||
initial_lambda: f32,
|
||||
min_lambda: f32,
|
||||
max_lambda: f32,
|
||||
fisher_ema_decay: f32,
|
||||
boundary_threshold: f32,
|
||||
gradient_history_size: usize,
|
||||
}
|
||||
|
||||
impl Genome {
|
||||
/// The shipped, hand-tuned SOTA baseline (mirrors `EwcConfig::default()`).
|
||||
fn baseline() -> Self {
|
||||
let d = EwcConfig::default();
|
||||
Self {
|
||||
lr: 0.1,
|
||||
initial_lambda: d.initial_lambda,
|
||||
min_lambda: d.min_lambda,
|
||||
max_lambda: d.max_lambda,
|
||||
fisher_ema_decay: d.fisher_ema_decay,
|
||||
boundary_threshold: d.boundary_threshold,
|
||||
gradient_history_size: d.gradient_history_size,
|
||||
}
|
||||
}
|
||||
|
||||
fn to_config(&self) -> EwcConfig {
|
||||
EwcConfig {
|
||||
param_count: PARAM_COUNT,
|
||||
max_tasks: 10,
|
||||
initial_lambda: self.initial_lambda,
|
||||
min_lambda: self.min_lambda,
|
||||
max_lambda: self.max_lambda,
|
||||
fisher_ema_decay: self.fisher_ema_decay,
|
||||
boundary_threshold: self.boundary_threshold,
|
||||
gradient_history_size: self.gradient_history_size,
|
||||
}
|
||||
}
|
||||
|
||||
fn clamp(&mut self) {
|
||||
self.lr = self.lr.clamp(0.01, 0.5);
|
||||
self.initial_lambda = self.initial_lambda.clamp(10.0, 20_000.0);
|
||||
self.min_lambda = self.min_lambda.clamp(1.0, 2_000.0);
|
||||
self.max_lambda = self.max_lambda.clamp(2_000.0, 50_000.0);
|
||||
self.fisher_ema_decay = self.fisher_ema_decay.clamp(0.90, 0.9999);
|
||||
self.boundary_threshold = self.boundary_threshold.clamp(0.5, 6.0);
|
||||
self.gradient_history_size = self.gradient_history_size.clamp(10, 200);
|
||||
}
|
||||
}
|
||||
|
||||
struct Metrics {
|
||||
avg_final_loss: f32,
|
||||
forgetting: f32,
|
||||
plasticity: f32,
|
||||
}
|
||||
|
||||
fn loss(w: &[f32], target: &[f32]) -> f32 {
|
||||
0.5 * w
|
||||
.iter()
|
||||
.zip(target)
|
||||
.map(|(a, b)| (a - b) * (a - b))
|
||||
.sum::<f32>()
|
||||
/ w.len() as f32
|
||||
}
|
||||
|
||||
/// Run one task-sequence (seeded) under a genome; return continual-learning metrics.
|
||||
fn run_sequence(g: &Genome, seq_seed: u64) -> Metrics {
|
||||
let p = PARAM_COUNT;
|
||||
let mut rng = StdRng::seed_from_u64(seq_seed);
|
||||
// Task targets: each task wants `w` near a different random point. Tasks share
|
||||
// the parameter space, so learning a later task drags `w` off earlier ones —
|
||||
// the forgetting pressure EWC must counter.
|
||||
let targets: Vec<Vec<f32>> = (0..N_TASKS)
|
||||
.map(|_| (0..p).map(|_| rng.gen_range(-1.0f32..1.0)).collect())
|
||||
.collect();
|
||||
|
||||
let mut ewc = EwcPlusPlus::new(g.to_config());
|
||||
let mut w = vec![0.0f32; p];
|
||||
let mut loss_just_after = vec![0.0f32; N_TASKS];
|
||||
|
||||
for (t, target) in targets.iter().enumerate() {
|
||||
for _ in 0..STEPS_PER_TASK {
|
||||
// Gradient of 0.5||w-target||^2 (the only signal — current task only).
|
||||
let grad: Vec<f32> = w.iter().zip(target).map(|(wi, ti)| wi - ti).collect();
|
||||
// The learner is NOT told task boundaries — it detects them itself.
|
||||
if ewc.detect_task_boundary(&grad) {
|
||||
ewc.start_new_task();
|
||||
}
|
||||
let gc = ewc.apply_constraints(&grad);
|
||||
for (wi, gi) in w.iter_mut().zip(gc.iter()) {
|
||||
*wi -= g.lr * gi;
|
||||
}
|
||||
ewc.update_fisher(&grad);
|
||||
ewc.set_optimal_weights(&w);
|
||||
}
|
||||
loss_just_after[t] = loss(&w, target); // plasticity probe (fresh)
|
||||
}
|
||||
|
||||
let final_losses: Vec<f32> = targets.iter().map(|tg| loss(&w, tg)).collect();
|
||||
let avg_final_loss = final_losses.iter().sum::<f32>() / N_TASKS as f32;
|
||||
// Forgetting: how much each task degraded between "just after" and "final".
|
||||
let forgetting = final_losses
|
||||
.iter()
|
||||
.zip(loss_just_after.iter())
|
||||
.map(|(f, a)| (f - a).max(0.0))
|
||||
.sum::<f32>()
|
||||
/ N_TASKS as f32;
|
||||
let plasticity = loss_just_after.iter().sum::<f32>() / N_TASKS as f32;
|
||||
Metrics {
|
||||
avg_final_loss,
|
||||
forgetting,
|
||||
plasticity,
|
||||
}
|
||||
}
|
||||
|
||||
/// Mean metrics over a set of seeds.
|
||||
fn evaluate(g: &Genome, seeds: &[u64]) -> Metrics {
|
||||
let mut af = 0.0;
|
||||
let mut fg = 0.0;
|
||||
let mut pl = 0.0;
|
||||
for &s in seeds {
|
||||
let m = run_sequence(g, s);
|
||||
af += m.avg_final_loss;
|
||||
fg += m.forgetting;
|
||||
pl += m.plasticity;
|
||||
}
|
||||
let n = seeds.len() as f32;
|
||||
Metrics {
|
||||
avg_final_loss: af / n,
|
||||
forgetting: fg / n,
|
||||
plasticity: pl / n,
|
||||
}
|
||||
}
|
||||
|
||||
/// Darwin fitness (higher = better): minimise final loss, penalise forgetting.
|
||||
fn fitness(g: &Genome, seeds: &[u64]) -> f32 {
|
||||
let m = evaluate(g, seeds);
|
||||
-(m.avg_final_loss + 0.3 * m.forgetting)
|
||||
}
|
||||
|
||||
fn mutate(g: &Genome, rng: &mut StdRng) -> Genome {
|
||||
let mut c = g.clone();
|
||||
if rng.gen::<f32>() < 0.5 {
|
||||
c.lr *= 0.6 + rng.gen::<f32>() * 0.8;
|
||||
}
|
||||
if rng.gen::<f32>() < 0.6 {
|
||||
c.initial_lambda *= 0.4 + rng.gen::<f32>() * 1.4;
|
||||
}
|
||||
if rng.gen::<f32>() < 0.4 {
|
||||
c.min_lambda *= 0.4 + rng.gen::<f32>() * 1.4;
|
||||
}
|
||||
if rng.gen::<f32>() < 0.4 {
|
||||
c.max_lambda *= 0.6 + rng.gen::<f32>() * 0.9;
|
||||
}
|
||||
if rng.gen::<f32>() < 0.4 {
|
||||
c.fisher_ema_decay += rng.gen::<f32>() * 0.04 - 0.02;
|
||||
}
|
||||
if rng.gen::<f32>() < 0.5 {
|
||||
c.boundary_threshold += rng.gen::<f32>() * 2.0 - 1.0;
|
||||
}
|
||||
if rng.gen::<f32>() < 0.3 {
|
||||
c.gradient_history_size =
|
||||
(c.gradient_history_size as i32 + rng.gen_range(-40..40)).max(10) as usize;
|
||||
}
|
||||
c.clamp();
|
||||
c
|
||||
}
|
||||
|
||||
fn main() {
|
||||
println!(
|
||||
"== SONA · Meta-Harness Darwin over EWC++ (freeze the algorithm, evolve the config) =="
|
||||
);
|
||||
println!(
|
||||
"continual learning: {N_TASKS} tasks × {STEPS_PER_TASK} steps, {PARAM_COUNT} params, auto task-boundary detection"
|
||||
);
|
||||
|
||||
let baseline = Genome::baseline();
|
||||
let base_train = evaluate(&baseline, TRAIN_SEEDS);
|
||||
println!(
|
||||
"\nbaseline (hand-tuned default λ={:.0}): train avg_final_loss {:.4} forgetting {:.4} plasticity {:.4}",
|
||||
baseline.initial_lambda, base_train.avg_final_loss, base_train.forgetting, base_train.plasticity
|
||||
);
|
||||
|
||||
// ── GA: evolve the genome on TRAIN seeds only ───────────────────────────────
|
||||
let mut rng = StdRng::seed_from_u64(0xE_C);
|
||||
const POP: usize = 24;
|
||||
const GEN: usize = 18;
|
||||
const ELITE: usize = 6;
|
||||
let mut pop: Vec<Genome> = std::iter::once(baseline.clone())
|
||||
.chain((0..POP - 1).map(|_| mutate(&baseline, &mut rng)))
|
||||
.collect();
|
||||
let mut best = (baseline.clone(), fitness(&baseline, TRAIN_SEEDS));
|
||||
|
||||
for gen in 0..GEN {
|
||||
// Reward-hacking guard (ADR-271): screen every candidate; non-finite or
|
||||
// degenerate (collapsed zero-loss) configs are EXCLUDED from the ranking
|
||||
// — not zero-scored — so a hack can neither win nor NaN-panic the sort.
|
||||
let guard = Guard::deterministic();
|
||||
let mut scored: Vec<(Genome, f32)> = Vec::new();
|
||||
let mut rejected = 0usize;
|
||||
for g in &pop {
|
||||
let f = fitness(g, TRAIN_SEEDS);
|
||||
let m = evaluate(g, TRAIN_SEEDS);
|
||||
let finite = f.is_finite() && m.avg_final_loss.is_finite() && m.forgetting.is_finite();
|
||||
match guard.screen(f, finite, true, m.avg_final_loss <= 0.0) {
|
||||
Verdict::Accepted(_) => scored.push((g.clone(), f)),
|
||||
Verdict::Rejected(_) => rejected += 1,
|
||||
}
|
||||
}
|
||||
if scored.is_empty() {
|
||||
scored.push(best.clone()); // never leave the population empty
|
||||
}
|
||||
scored.sort_by(|a, b| b.1.partial_cmp(&a.1).unwrap_or(std::cmp::Ordering::Equal));
|
||||
if scored[0].1 > best.1 {
|
||||
best = scored[0].clone();
|
||||
}
|
||||
let _ = rejected;
|
||||
if gen % 3 == 0 || gen == GEN - 1 {
|
||||
let m = evaluate(&scored[0].0, TRAIN_SEEDS);
|
||||
println!(
|
||||
"gen {gen:2}: train avg_final_loss {:.4} forgetting {:.4} (λ0={:.0} lr={:.3} bθ={:.2}) fitness {:.4}",
|
||||
m.avg_final_loss, m.forgetting, scored[0].0.initial_lambda, scored[0].0.lr, scored[0].0.boundary_threshold, scored[0].1
|
||||
);
|
||||
}
|
||||
let elites: Vec<Genome> = scored.iter().take(ELITE).map(|(g, _)| g.clone()).collect();
|
||||
let mut next = elites.clone();
|
||||
while next.len() < POP {
|
||||
let parent = &elites[rng.gen_range(0..elites.len())];
|
||||
next.push(mutate(parent, &mut rng));
|
||||
}
|
||||
pop = next;
|
||||
}
|
||||
|
||||
// ── Coordinate-descent polish (deterministic, reproducible optimum) ─────────
|
||||
let evolved = polish(best.0.clone());
|
||||
|
||||
// ── Report on HELD-OUT test seeds (never seen during evolution) ─────────────
|
||||
let base_test = evaluate(&baseline, TEST_SEEDS);
|
||||
let evo_test = evaluate(&evolved, TEST_SEEDS);
|
||||
let evo_train = evaluate(&evolved, TRAIN_SEEDS);
|
||||
|
||||
println!("\n-- evolved genome --");
|
||||
println!(
|
||||
" λ0={:.0} λmin={:.0} λmax={:.0} decay={:.4} bθ={:.2} hist={} lr={:.3}",
|
||||
evolved.initial_lambda,
|
||||
evolved.min_lambda,
|
||||
evolved.max_lambda,
|
||||
evolved.fisher_ema_decay,
|
||||
evolved.boundary_threshold,
|
||||
evolved.gradient_history_size,
|
||||
evolved.lr
|
||||
);
|
||||
println!(
|
||||
" train: avg_final_loss {:.4} (baseline {:.4})",
|
||||
evo_train.avg_final_loss, base_train.avg_final_loss
|
||||
);
|
||||
|
||||
println!("\n-- HELD-OUT test sequences (the beyond-SOTA result) --");
|
||||
println!(
|
||||
" baseline: avg_final_loss {:.4} forgetting {:.4} plasticity {:.4}",
|
||||
base_test.avg_final_loss, base_test.forgetting, base_test.plasticity
|
||||
);
|
||||
println!(
|
||||
" evolved : avg_final_loss {:.4} forgetting {:.4} plasticity {:.4}",
|
||||
evo_test.avg_final_loss, evo_test.forgetting, evo_test.plasticity
|
||||
);
|
||||
let gain =
|
||||
(base_test.avg_final_loss - evo_test.avg_final_loss) / base_test.avg_final_loss * 100.0;
|
||||
let fgain =
|
||||
(base_test.forgetting - evo_test.forgetting) / base_test.forgetting.max(1e-9) * 100.0;
|
||||
println!(
|
||||
" → {:.1}% lower final loss, {:.1}% less forgetting on UNSEEN task sequences",
|
||||
gain, fgain
|
||||
);
|
||||
println!(
|
||||
"{}",
|
||||
if evo_test.avg_final_loss < base_test.avg_final_loss {
|
||||
"BEYOND SOTA — metaharness-evolved EWC++ config beats the hand-tuned default on held-out tasks"
|
||||
} else {
|
||||
"no improvement over baseline this run"
|
||||
}
|
||||
);
|
||||
}
|
||||
|
||||
/// Greedy per-gene coordinate descent over a candidate grid (cross-seed mean on
|
||||
/// TRAIN) — converts the GA's broad search into a reproducible optimum.
|
||||
fn polish(seed: Genome) -> Genome {
|
||||
let mut cur = seed;
|
||||
let mut cur_f = fitness(&cur, TRAIN_SEEDS);
|
||||
let lambdas: [f32; 8] = [50.0, 200.0, 500.0, 1000.0, 2000.0, 4000.0, 8000.0, 15000.0];
|
||||
let lrs: [f32; 6] = [0.02, 0.05, 0.1, 0.15, 0.2, 0.3];
|
||||
let bthr: [f32; 6] = [0.8, 1.5, 2.0, 3.0, 4.0, 5.0];
|
||||
let decays: [f32; 4] = [0.95, 0.99, 0.999, 0.9999];
|
||||
for _ in 0..3 {
|
||||
let mut improved = false;
|
||||
macro_rules! try_gene {
|
||||
($field:ident, $cands:expr) => {
|
||||
for &v in $cands.iter() {
|
||||
if (cur.$field - v).abs() < f32::EPSILON {
|
||||
continue;
|
||||
}
|
||||
let mut cand = cur.clone();
|
||||
cand.$field = v;
|
||||
cand.clamp();
|
||||
let f = fitness(&cand, TRAIN_SEEDS);
|
||||
if f > cur_f + 1e-9 {
|
||||
cur = cand;
|
||||
cur_f = f;
|
||||
improved = true;
|
||||
}
|
||||
}
|
||||
};
|
||||
}
|
||||
try_gene!(initial_lambda, lambdas);
|
||||
try_gene!(lr, lrs);
|
||||
try_gene!(boundary_threshold, bthr);
|
||||
try_gene!(fisher_ema_decay, decays);
|
||||
if !improved {
|
||||
break;
|
||||
}
|
||||
}
|
||||
cur
|
||||
}
|
||||
284
crates/sona/examples/darwin_router.rs
Normal file
284
crates/sona/examples/darwin_router.rs
Normal file
|
|
@ -0,0 +1,284 @@
|
|||
//! Per-task-category genome router (ADR-271, Ornith-1.0 borrow #2).
|
||||
//!
|
||||
//! Ornith-1.0's main empirical result is that **per-task-category strategies
|
||||
//! emerge** — no single scaffold is optimal across workload types. The metaharness
|
||||
//! analogue: instead of evolving ONE global `EwcConfig`, evolve a **router**
|
||||
//! `task-class → genome`, so each workload class gets its own specialized config.
|
||||
//!
|
||||
//! Two continual-learning workload classes with *conflicting* optima:
|
||||
//! * `STABLE` — few tasks, long training, small shifts → wants to *learn*
|
||||
//! (low lambda; aggressive EWC protection only wastes plasticity).
|
||||
//! * `VOLATILE` — many tasks, short training, large shifts → wants to *retain*
|
||||
//! (high lambda; without it, later tasks erase earlier ones).
|
||||
//!
|
||||
//! Baseline = the single best global genome (the PR-#615 approach, evolved over
|
||||
//! both classes pooled). Router = the best genome PER class. We report both on
|
||||
//! HELD-OUT sequences. Beyond-SOTA result: the router beats the single global
|
||||
//! genome on unseen sequences — because one config cannot serve conflicting
|
||||
//! workloads. Selection is screened by `darwin_guard`.
|
||||
//!
|
||||
//! Run: `cargo run -p ruvector-sona --release --example darwin_router`
|
||||
|
||||
use rand::rngs::StdRng;
|
||||
use rand::{Rng, SeedableRng};
|
||||
use ruvector_sona::darwin_guard::{Guard, Verdict};
|
||||
use ruvector_sona::{EwcConfig, EwcPlusPlus};
|
||||
|
||||
const PARAM_COUNT: usize = 96;
|
||||
// Enough per-class data that specialization generalizes (Ornith's regime: many
|
||||
// per-category samples) rather than overfitting a handful of sequences.
|
||||
const TRAIN_SEEDS: &[u64] = &[1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16];
|
||||
const TEST_SEEDS: &[u64] = &[101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112]; // held out
|
||||
|
||||
/// A workload class (task-category).
|
||||
#[derive(Clone, Copy)]
|
||||
struct Class {
|
||||
name: &'static str,
|
||||
n_tasks: usize,
|
||||
steps: usize,
|
||||
shift: f32, // inter-task target magnitude (forgetting pressure)
|
||||
}
|
||||
const STABLE: Class = Class {
|
||||
name: "STABLE",
|
||||
n_tasks: 3,
|
||||
steps: 120,
|
||||
shift: 0.25,
|
||||
};
|
||||
const VOLATILE: Class = Class {
|
||||
name: "VOLATILE",
|
||||
n_tasks: 9,
|
||||
steps: 35,
|
||||
shift: 1.2,
|
||||
};
|
||||
const CLASSES: [Class; 2] = [STABLE, VOLATILE];
|
||||
|
||||
#[derive(Clone, Debug)]
|
||||
struct Genome {
|
||||
lr: f32,
|
||||
initial_lambda: f32,
|
||||
max_lambda: f32,
|
||||
fisher_ema_decay: f32,
|
||||
boundary_threshold: f32,
|
||||
}
|
||||
impl Genome {
|
||||
fn baseline() -> Self {
|
||||
let d = EwcConfig::default();
|
||||
Self {
|
||||
lr: 0.1,
|
||||
initial_lambda: d.initial_lambda,
|
||||
max_lambda: d.max_lambda,
|
||||
fisher_ema_decay: d.fisher_ema_decay,
|
||||
boundary_threshold: d.boundary_threshold,
|
||||
}
|
||||
}
|
||||
fn to_config(&self) -> EwcConfig {
|
||||
EwcConfig {
|
||||
param_count: PARAM_COUNT,
|
||||
max_tasks: 12,
|
||||
initial_lambda: self.initial_lambda,
|
||||
min_lambda: 50.0,
|
||||
max_lambda: self.max_lambda,
|
||||
fisher_ema_decay: self.fisher_ema_decay,
|
||||
boundary_threshold: self.boundary_threshold,
|
||||
gradient_history_size: 60,
|
||||
}
|
||||
}
|
||||
fn clamp(&mut self) {
|
||||
self.lr = self.lr.clamp(0.01, 0.5);
|
||||
self.initial_lambda = self.initial_lambda.clamp(10.0, 20_000.0);
|
||||
self.max_lambda = self.max_lambda.clamp(2_000.0, 50_000.0);
|
||||
self.fisher_ema_decay = self.fisher_ema_decay.clamp(0.90, 0.9999);
|
||||
self.boundary_threshold = self.boundary_threshold.clamp(0.5, 6.0);
|
||||
}
|
||||
}
|
||||
|
||||
fn run_sequence(class: &Class, g: &Genome, seed: u64) -> f32 {
|
||||
let p = PARAM_COUNT;
|
||||
let mut rng = StdRng::seed_from_u64(seed ^ (class.n_tasks as u64).wrapping_mul(0x9E37));
|
||||
let targets: Vec<Vec<f32>> = (0..class.n_tasks)
|
||||
.map(|_| {
|
||||
(0..p)
|
||||
.map(|_| class.shift * rng.gen_range(-1.0f32..1.0))
|
||||
.collect()
|
||||
})
|
||||
.collect();
|
||||
let mut ewc = EwcPlusPlus::new(g.to_config());
|
||||
let mut w = vec![0.0f32; p];
|
||||
for target in &targets {
|
||||
for _ in 0..class.steps {
|
||||
let grad: Vec<f32> = w.iter().zip(target).map(|(wi, ti)| wi - ti).collect();
|
||||
if ewc.detect_task_boundary(&grad) {
|
||||
ewc.start_new_task();
|
||||
}
|
||||
let gc = ewc.apply_constraints(&grad);
|
||||
for (wi, gi) in w.iter_mut().zip(gc.iter()) {
|
||||
*wi -= g.lr * gi;
|
||||
}
|
||||
ewc.update_fisher(&grad);
|
||||
ewc.set_optimal_weights(&w);
|
||||
}
|
||||
}
|
||||
targets
|
||||
.iter()
|
||||
.map(|tg| {
|
||||
0.5 * w
|
||||
.iter()
|
||||
.zip(tg)
|
||||
.map(|(a, b)| (a - b) * (a - b))
|
||||
.sum::<f32>()
|
||||
/ p as f32
|
||||
})
|
||||
.sum::<f32>()
|
||||
/ class.n_tasks as f32
|
||||
}
|
||||
|
||||
fn eval_class(class: &Class, g: &Genome, seeds: &[u64]) -> f32 {
|
||||
seeds
|
||||
.iter()
|
||||
.map(|&s| run_sequence(class, g, s))
|
||||
.sum::<f32>()
|
||||
/ seeds.len() as f32
|
||||
}
|
||||
|
||||
fn mutate(g: &Genome, rng: &mut StdRng) -> Genome {
|
||||
let mut c = g.clone();
|
||||
if rng.gen::<f32>() < 0.5 {
|
||||
c.lr *= 0.6 + rng.gen::<f32>() * 0.8;
|
||||
}
|
||||
if rng.gen::<f32>() < 0.6 {
|
||||
c.initial_lambda *= 0.35 + rng.gen::<f32>() * 1.5;
|
||||
}
|
||||
if rng.gen::<f32>() < 0.5 {
|
||||
c.max_lambda *= 0.6 + rng.gen::<f32>() * 0.9;
|
||||
}
|
||||
if rng.gen::<f32>() < 0.4 {
|
||||
c.fisher_ema_decay += rng.gen::<f32>() * 0.04 - 0.02;
|
||||
}
|
||||
if rng.gen::<f32>() < 0.5 {
|
||||
c.boundary_threshold += rng.gen::<f32>() * 2.0 - 1.0;
|
||||
}
|
||||
c.clamp();
|
||||
c
|
||||
}
|
||||
|
||||
/// Evolve a genome minimising `fit` (a loss closure), guard-screened.
|
||||
fn evolve(seed: u64, fit: &dyn Fn(&Genome) -> f32) -> Genome {
|
||||
let guard = Guard::deterministic();
|
||||
let mut rng = StdRng::seed_from_u64(seed);
|
||||
let base = Genome::baseline();
|
||||
const POP: usize = 18;
|
||||
const GEN: usize = 14;
|
||||
let mut pop: Vec<Genome> = std::iter::once(base.clone())
|
||||
.chain((0..POP - 1).map(|_| mutate(&base, &mut rng)))
|
||||
.collect();
|
||||
let mut best = (base.clone(), fit(&base));
|
||||
for _ in 0..GEN {
|
||||
let mut scored: Vec<(Genome, f32)> = Vec::new();
|
||||
for g in &pop {
|
||||
let loss = fit(g);
|
||||
// Guard: exclude non-finite / degenerate (negative loss is impossible here).
|
||||
match guard.screen(-loss, loss.is_finite(), true, loss < 0.0) {
|
||||
Verdict::Accepted(_) => scored.push((g.clone(), loss)),
|
||||
Verdict::Rejected(_) => {}
|
||||
}
|
||||
}
|
||||
if scored.is_empty() {
|
||||
scored.push(best.clone());
|
||||
}
|
||||
scored.sort_by(|a, b| a.1.partial_cmp(&b.1).unwrap_or(std::cmp::Ordering::Equal));
|
||||
if scored[0].1 < best.1 {
|
||||
best = scored[0].clone();
|
||||
}
|
||||
let elites: Vec<Genome> = scored.iter().take(5).map(|(g, _)| g.clone()).collect();
|
||||
let mut next = elites.clone();
|
||||
while next.len() < POP {
|
||||
next.push(mutate(&elites[rng.gen_range(0..elites.len())], &mut rng));
|
||||
}
|
||||
pop = next;
|
||||
}
|
||||
best.0
|
||||
}
|
||||
|
||||
fn main() {
|
||||
println!("== SONA · per-task-category genome ROUTER (ADR-271, Ornith borrow) ==");
|
||||
let base = Genome::baseline();
|
||||
for c in &CLASSES {
|
||||
println!(
|
||||
"class {:8} baseline held-out loss {:.4}",
|
||||
c.name,
|
||||
eval_class(c, &base, TEST_SEEDS)
|
||||
);
|
||||
}
|
||||
|
||||
// Single GLOBAL genome (PR #615 approach): one config over BOTH classes pooled.
|
||||
let global = evolve(0xA10BA1, &|g| {
|
||||
CLASSES
|
||||
.iter()
|
||||
.map(|c| eval_class(c, g, TRAIN_SEEDS))
|
||||
.sum::<f32>()
|
||||
/ CLASSES.len() as f32
|
||||
});
|
||||
|
||||
// ROUTER: one specialized genome PER class.
|
||||
let router: Vec<(Class, Genome)> = CLASSES
|
||||
.iter()
|
||||
.map(|c| {
|
||||
(
|
||||
*c,
|
||||
evolve(0x12345 ^ c.n_tasks as u64, &|g| {
|
||||
eval_class(c, g, TRAIN_SEEDS)
|
||||
}),
|
||||
)
|
||||
})
|
||||
.collect();
|
||||
|
||||
// Held-out comparison.
|
||||
let global_loss = CLASSES
|
||||
.iter()
|
||||
.map(|c| eval_class(c, &global, TEST_SEEDS))
|
||||
.sum::<f32>()
|
||||
/ CLASSES.len() as f32;
|
||||
let router_loss = router
|
||||
.iter()
|
||||
.map(|(c, g)| eval_class(c, g, TEST_SEEDS))
|
||||
.sum::<f32>()
|
||||
/ router.len() as f32;
|
||||
|
||||
println!("\n-- per-class evolved configs --");
|
||||
println!(
|
||||
" GLOBAL (one config): λ0={:.0} lr={:.3} bθ={:.2}",
|
||||
global.initial_lambda, global.lr, global.boundary_threshold
|
||||
);
|
||||
for (c, g) in &router {
|
||||
println!(
|
||||
" {:8} (router) : λ0={:.0} lr={:.3} bθ={:.2} held-out {:.4}",
|
||||
c.name,
|
||||
g.initial_lambda,
|
||||
g.lr,
|
||||
g.boundary_threshold,
|
||||
eval_class(c, g, TEST_SEEDS)
|
||||
);
|
||||
}
|
||||
|
||||
println!("\n-- HELD-OUT (mean over classes) --");
|
||||
println!(" single global genome (PR #615): {global_loss:.4}");
|
||||
println!(" per-category router : {router_loss:.4}");
|
||||
let gain = (global_loss - router_loss) / global_loss * 100.0;
|
||||
println!(
|
||||
" → router is {gain:.1}% better on held-out (specialization beats one-config-fits-all)"
|
||||
);
|
||||
println!(
|
||||
"{}",
|
||||
if router_loss < global_loss - 1e-4 {
|
||||
"BEYOND SOTA — per-task-category routing beats the single best global config on unseen sequences"
|
||||
} else {
|
||||
"no improvement over the single global genome this run"
|
||||
}
|
||||
);
|
||||
println!(
|
||||
" (caveat: the gain is the value of specialization — modest here, and it REVERSES\n \
|
||||
when per-class data is scarce: a specialized config then overfits while the pooled\n \
|
||||
global generalizes. Per-category routing needs enough per-category samples — Ornith's regime.)"
|
||||
);
|
||||
assert!(router_loss.is_finite() && global_loss.is_finite());
|
||||
}
|
||||
235
crates/sona/examples/darwin_weightadapter.rs
Normal file
235
crates/sona/examples/darwin_weightadapter.rs
Normal file
|
|
@ -0,0 +1,235 @@
|
|||
//! The `weightAdapter` gene — Darwin selects a fine-tuned LoRA adapter the way it
|
||||
//! selects any other gene, and *empirically prunes* adapters that regress.
|
||||
//!
|
||||
//! Premise (the "autonomous data engine"): a fine-tune produces a candidate
|
||||
//! adapter delta `Δw` (e.g. a LoRA distilled from verified SWE-bench trajectories).
|
||||
//! Instead of *assuming* the new weights are better, expose the adapter as a gene
|
||||
//! `(which_adapter, alpha)` and let evolutionary selection decide whether — and
|
||||
//! how much — to apply it (`w_eff = w_base + alpha·Δw`), scored by held-out task
|
||||
//! performance. A fine-tune that overfits gets pruned by selection.
|
||||
//!
|
||||
//! ## The subtlety this demonstrates (and why it matters)
|
||||
//!
|
||||
//! "Selection prunes overfit adapters" is TRUE — **but only if the fitness is
|
||||
//! evaluated per-domain.** With a single *aggregate* fitness, an adapter whose
|
||||
//! in-distribution gain outweighs its out-of-distribution loss is *selected* and
|
||||
//! silently regresses the out-of-dist domain. Only a **per-domain / no-regression
|
||||
//! (Pareto)** rule — "must not regress ANY repository" — actually prunes it.
|
||||
//!
|
||||
//! Two candidate adapters, distilled from TRAIN sequences:
|
||||
//! * `general` — captures the signal common to both domains → helps both.
|
||||
//! * `overfit` — captures domain-A-specific structure → helps A, hurts B.
|
||||
//!
|
||||
//! We then pick the best `(adapter, alpha)` under each selection rule on HELD-OUT
|
||||
//! sequences and show: aggregate accepts `overfit` (and regresses B); per-domain
|
||||
//! prunes it and keeps `general`.
|
||||
//!
|
||||
//! Run: `cargo run -p ruvector-sona --release --example darwin_weightadapter`
|
||||
|
||||
use rand::rngs::StdRng;
|
||||
use rand::{Rng, SeedableRng};
|
||||
|
||||
const DIM: usize = 64;
|
||||
const TRAIN_SEEDS: &[u64] = &[1, 2, 3, 4, 5, 6, 7, 8];
|
||||
// Held-out eval is IMBALANCED toward in-dist (A) — the realistic case: the eval
|
||||
// pool is dominated by the same repos the adapter was fine-tuned on. A naive
|
||||
// (volume-weighted) aggregate fitness is therefore easy to fool.
|
||||
const TEST_A: &[u64] = &[101, 102, 103, 104, 105, 106]; // in-dist (majority)
|
||||
const TEST_B: &[u64] = &[201, 202]; // out-of-dist (minority)
|
||||
const NA: f32 = 6.0;
|
||||
const NB: f32 = 2.0;
|
||||
|
||||
#[derive(Clone, Copy, PartialEq, Debug)]
|
||||
enum Adapter {
|
||||
None,
|
||||
General,
|
||||
Overfit,
|
||||
}
|
||||
|
||||
/// Fixed structure of the two domains: a shared signal + per-domain offsets.
|
||||
struct World {
|
||||
mu_common: Vec<f32>,
|
||||
delta_a: Vec<f32>, // domain-A-specific direction
|
||||
delta_b: Vec<f32>, // domain-B-specific direction
|
||||
}
|
||||
|
||||
impl World {
|
||||
fn new() -> Self {
|
||||
let mut rng = StdRng::seed_from_u64(0xC0FFEE);
|
||||
let rv = |rng: &mut StdRng| {
|
||||
(0..DIM)
|
||||
.map(|_| rng.gen_range(-1.0f32..1.0))
|
||||
.collect::<Vec<_>>()
|
||||
};
|
||||
let delta_a = rv(&mut rng);
|
||||
// Domain B's structure OPPOSES domain A's — a small shared signal plus
|
||||
// conflicting domain-specific directions, so an A-overfit adapter that
|
||||
// captures `delta_a` actively hurts B (the realistic "fine-tune helped
|
||||
// some repos, regressed others" case).
|
||||
let delta_b: Vec<f32> = delta_a.iter().map(|x| -x).collect();
|
||||
let mu_common: Vec<f32> = rv(&mut rng).iter().map(|x| 0.3 * x).collect(); // weak shared signal
|
||||
Self {
|
||||
mu_common,
|
||||
delta_a,
|
||||
delta_b,
|
||||
}
|
||||
}
|
||||
|
||||
/// A target for domain `a_side` (true = domain A / in-dist, false = B / out-dist).
|
||||
fn target(&self, a_side: bool, seed: u64) -> Vec<f32> {
|
||||
let mut rng = StdRng::seed_from_u64(seed ^ if a_side { 0xA } else { 0xB });
|
||||
let d = if a_side { &self.delta_a } else { &self.delta_b };
|
||||
(0..DIM)
|
||||
.map(|i| self.mu_common[i] + d[i] + 0.05 * rng.gen_range(-1.0f32..1.0))
|
||||
.collect()
|
||||
}
|
||||
}
|
||||
|
||||
fn loss(w: &[f32], target: &[f32]) -> f32 {
|
||||
w.iter()
|
||||
.zip(target)
|
||||
.map(|(a, b)| (a - b) * (a - b))
|
||||
.sum::<f32>()
|
||||
/ w.len() as f32
|
||||
}
|
||||
|
||||
/// Distil an adapter delta from TRAIN targets (mean target = the "fine-tune").
|
||||
/// `general` averages both domains (shared signal); `overfit` uses domain A only.
|
||||
fn distil(world: &World, adapter: Adapter) -> Vec<f32> {
|
||||
match adapter {
|
||||
Adapter::None => vec![0.0; DIM],
|
||||
Adapter::Overfit => mean_target(world, &[true], TRAIN_SEEDS),
|
||||
Adapter::General => mean_target(world, &[true, false], TRAIN_SEEDS),
|
||||
}
|
||||
}
|
||||
|
||||
fn mean_target(world: &World, sides: &[bool], seeds: &[u64]) -> Vec<f32> {
|
||||
let mut acc = vec![0.0f32; DIM];
|
||||
let mut n = 0.0;
|
||||
for &side in sides {
|
||||
for &s in seeds {
|
||||
let t = world.target(side, s);
|
||||
for (a, ti) in acc.iter_mut().zip(t.iter()) {
|
||||
*a += ti;
|
||||
}
|
||||
n += 1.0;
|
||||
}
|
||||
}
|
||||
acc.iter().map(|x| x / n).collect()
|
||||
}
|
||||
|
||||
/// Mean task loss on a domain (seeds) with `w_eff = alpha·Δw` (base w = 0: the
|
||||
/// adapter supplies all signal — what matters here is *relative* improvement).
|
||||
fn domain_loss(world: &World, a_side: bool, delta: &[f32], alpha: f32, seeds: &[u64]) -> f32 {
|
||||
let w: Vec<f32> = delta.iter().map(|d| alpha * d).collect();
|
||||
seeds
|
||||
.iter()
|
||||
.map(|&s| loss(&w, &world.target(a_side, s)))
|
||||
.sum::<f32>()
|
||||
/ seeds.len() as f32
|
||||
}
|
||||
|
||||
fn main() {
|
||||
let world = World::new();
|
||||
let alphas: Vec<f32> = (0..=20).map(|i| i as f32 / 20.0).collect();
|
||||
let candidates = [Adapter::None, Adapter::General, Adapter::Overfit];
|
||||
|
||||
// Base (no adapter) reference loss per domain, on held-out seeds.
|
||||
let base_a = domain_loss(&world, true, &vec![0.0; DIM], 0.0, TEST_A);
|
||||
let base_b = domain_loss(&world, false, &vec![0.0; DIM], 0.0, TEST_B);
|
||||
|
||||
println!("== SONA · the `weightAdapter` gene — Darwin selects/prunes a fine-tuned adapter ==");
|
||||
println!("two domains (A = in-dist, B = out-dist); base (no adapter) held-out loss: A {base_a:.4} B {base_b:.4}\n");
|
||||
|
||||
// Evaluate every (adapter, alpha) on held-out seeds → improvement per domain.
|
||||
struct Cand {
|
||||
adapter: Adapter,
|
||||
alpha: f32,
|
||||
gain_a: f32,
|
||||
gain_b: f32,
|
||||
}
|
||||
let mut grid = Vec::new();
|
||||
for &adapter in &candidates {
|
||||
let delta = distil(&world, adapter);
|
||||
for &alpha in &alphas {
|
||||
let la = domain_loss(&world, true, &delta, alpha, TEST_A);
|
||||
let lb = domain_loss(&world, false, &delta, alpha, TEST_B);
|
||||
grid.push(Cand {
|
||||
adapter,
|
||||
alpha,
|
||||
gain_a: base_a - la, // >0 = improves domain A
|
||||
gain_b: base_b - lb, // >0 = improves domain B
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
// Volume-weighted (pooled) aggregate fitness — in-dist dominates the eval.
|
||||
let pooled = |c: &Cand| (NA * c.gain_a + NB * c.gain_b) / (NA + NB);
|
||||
|
||||
// Per-adapter best (illustrate the overfit asymmetry).
|
||||
println!("-- best alpha per adapter (held-out) --");
|
||||
for &adapter in &candidates {
|
||||
if adapter == Adapter::None {
|
||||
continue;
|
||||
}
|
||||
let best = grid
|
||||
.iter()
|
||||
.filter(|c| c.adapter == adapter)
|
||||
.max_by(|x, y| pooled(x).partial_cmp(&pooled(y)).unwrap())
|
||||
.unwrap();
|
||||
println!(
|
||||
" {:8?}: α={:.2} ΔA {:+.4} ΔB {:+.4}",
|
||||
adapter, best.alpha, best.gain_a, best.gain_b
|
||||
);
|
||||
}
|
||||
|
||||
// ── Selection rule 1: AGGREGATE fitness (mean gain) ─────────────────────────
|
||||
let agg = grid
|
||||
.iter()
|
||||
.max_by(|x, y| pooled(x).partial_cmp(&pooled(y)).unwrap())
|
||||
.unwrap();
|
||||
// ── Selection rule 2: PER-DOMAIN no-regression (Pareto) ─────────────────────
|
||||
// Accept only candidates that do NOT regress either domain, then maximise gain.
|
||||
let perdomain = grid
|
||||
.iter()
|
||||
.filter(|c| c.gain_a >= -1e-4 && c.gain_b >= -1e-4)
|
||||
.max_by(|x, y| pooled(x).partial_cmp(&pooled(y)).unwrap())
|
||||
.unwrap();
|
||||
|
||||
println!("\n-- selection outcome --");
|
||||
println!(
|
||||
" AGGREGATE fitness → picks {:8?} α={:.2}: ΔA {:+.4} ΔB {:+.4} {}",
|
||||
agg.adapter,
|
||||
agg.alpha,
|
||||
agg.gain_a,
|
||||
agg.gain_b,
|
||||
if agg.gain_b < -1e-4 {
|
||||
"← REGRESSES domain B (silently accepted)"
|
||||
} else {
|
||||
""
|
||||
}
|
||||
);
|
||||
println!(
|
||||
" PER-DOMAIN (Pareto)→ picks {:8?} α={:.2}: ΔA {:+.4} ΔB {:+.4} {}",
|
||||
perdomain.adapter,
|
||||
perdomain.alpha,
|
||||
perdomain.gain_a,
|
||||
perdomain.gain_b,
|
||||
if perdomain.adapter != agg.adapter {
|
||||
"← pruned the overfit adapter"
|
||||
} else {
|
||||
""
|
||||
}
|
||||
);
|
||||
|
||||
println!("\n-- conclusion --");
|
||||
println!("The weightAdapter gene lets Darwin keep a generalizing fine-tune AND reject an");
|
||||
println!("overfit one — but ONLY under per-domain (no-regression) selection. A single");
|
||||
println!("aggregate fitness is fooled by an adapter whose in-dist gain hides an out-dist");
|
||||
println!("regression. Evolve the adapter as a gene, but score it per-repository.");
|
||||
|
||||
assert!(
|
||||
perdomain.gain_a >= -1e-4 && perdomain.gain_b >= -1e-4,
|
||||
"Pareto pick must not regress"
|
||||
);
|
||||
}
|
||||
199
crates/sona/src/auto_tuner.rs
Normal file
199
crates/sona/src/auto_tuner.rs
Normal file
|
|
@ -0,0 +1,199 @@
|
|||
//! Online auto-tuner machinery for SONA config (ADR-271, Ornith-1.0 borrow #4).
|
||||
//!
|
||||
//! Offline evolution tunes a config to a *fixed* benchmark. Real workloads drift
|
||||
//! (non-stationary trajectory streams), so a fixed config goes stale. This module
|
||||
//! provides the **staleness-weighted** primitives for an *online* tuner that
|
||||
//! re-optimizes against the live stream, weighting recent observations over old
|
||||
//! ones via Ornith-1.0's staleness weight `w(d_t)`:
|
||||
//!
|
||||
//! ```text
|
||||
//! w(d) = 1 if d <= k1 (fresh — full weight)
|
||||
//! = exp(-lambda*(d - k1)) if k1 < d <= k2 (decaying)
|
||||
//! = 0 if d > k2 (too stale — dropped)
|
||||
//! ```
|
||||
//!
|
||||
//! where `d` is the *age* (clock ticks since the observation). The
|
||||
//! [`StalenessWindow`] maintains a staleness-weighted running estimate of "how
|
||||
//! well the current config is doing lately"; a `(1+1)`-ES on top of it (see
|
||||
//! `examples/darwin_autotuner.rs`) accepts a perturbed config only when its
|
||||
//! recent, freshness-weighted score beats the incumbent — so the tuner tracks a
|
||||
//! drifting optimum instead of averaging over a stale past.
|
||||
|
||||
use std::collections::VecDeque;
|
||||
|
||||
/// Ornith-1.0 staleness schedule `w(d_t)`.
|
||||
#[derive(Clone, Copy, Debug)]
|
||||
pub struct StalenessSchedule {
|
||||
/// Ages `<= k1` keep full weight 1.0.
|
||||
pub k1: u64,
|
||||
/// Ages `> k2` are dropped (weight 0).
|
||||
pub k2: u64,
|
||||
/// Exponential decay rate in the `(k1, k2]` band.
|
||||
pub lambda: f32,
|
||||
}
|
||||
|
||||
impl StalenessSchedule {
|
||||
/// A sensible default: full weight for 16 ticks, decay to ~0 by 64.
|
||||
#[must_use]
|
||||
pub fn new(k1: u64, k2: u64, lambda: f32) -> Self {
|
||||
Self { k1, k2, lambda }
|
||||
}
|
||||
|
||||
/// `w(d)` for an observation of age `d` ticks.
|
||||
#[must_use]
|
||||
pub fn weight(&self, age: u64) -> f32 {
|
||||
if age <= self.k1 {
|
||||
1.0
|
||||
} else if age <= self.k2 {
|
||||
(-self.lambda * (age - self.k1) as f32).exp()
|
||||
} else {
|
||||
0.0
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
impl Default for StalenessSchedule {
|
||||
fn default() -> Self {
|
||||
Self {
|
||||
k1: 16,
|
||||
k2: 64,
|
||||
lambda: 0.08,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// A staleness-weighted window of recent scalar observations (e.g. per-step loss
|
||||
/// under the current config). `push` advances the clock; `weighted_mean` reports
|
||||
/// the freshness-weighted average; observations past `k2` are evicted.
|
||||
#[derive(Clone, Debug)]
|
||||
pub struct StalenessWindow {
|
||||
schedule: StalenessSchedule,
|
||||
/// `(value, recorded_at_clock)` newest-last.
|
||||
samples: VecDeque<(f32, u64)>,
|
||||
clock: u64,
|
||||
cap: usize,
|
||||
}
|
||||
|
||||
impl StalenessWindow {
|
||||
/// New window with the given schedule and a hard capacity cap.
|
||||
#[must_use]
|
||||
pub fn new(schedule: StalenessSchedule, cap: usize) -> Self {
|
||||
Self {
|
||||
schedule,
|
||||
samples: VecDeque::with_capacity(cap),
|
||||
clock: 0,
|
||||
cap: cap.max(1),
|
||||
}
|
||||
}
|
||||
|
||||
/// Record an observation under the current config; advances the clock and
|
||||
/// evicts samples that are too stale (`age > k2`) or over capacity.
|
||||
pub fn push(&mut self, value: f32) {
|
||||
self.samples.push_back((value, self.clock));
|
||||
self.clock += 1;
|
||||
while self.samples.len() > self.cap {
|
||||
self.samples.pop_front();
|
||||
}
|
||||
// Evict fully-stale observations from the front.
|
||||
while let Some(&(_, t)) = self.samples.front() {
|
||||
if self.clock.saturating_sub(t) > self.schedule.k2 {
|
||||
self.samples.pop_front();
|
||||
} else {
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Reset the recorded observations but keep the clock running — used after a
|
||||
/// config switch so the new config is scored on *its own* fresh samples.
|
||||
pub fn clear_samples(&mut self) {
|
||||
self.samples.clear();
|
||||
}
|
||||
|
||||
/// Staleness-weighted mean of the window, or `None` if empty / all-stale.
|
||||
#[must_use]
|
||||
pub fn weighted_mean(&self) -> Option<f32> {
|
||||
let mut num = 0.0f32;
|
||||
let mut den = 0.0f32;
|
||||
for &(v, t) in &self.samples {
|
||||
let w = self.schedule.weight(self.clock.saturating_sub(t));
|
||||
num += w * v;
|
||||
den += w;
|
||||
}
|
||||
if den > 0.0 {
|
||||
Some(num / den)
|
||||
} else {
|
||||
None
|
||||
}
|
||||
}
|
||||
|
||||
/// Current clock (number of observations ever pushed).
|
||||
#[must_use]
|
||||
pub fn clock(&self) -> u64 {
|
||||
self.clock
|
||||
}
|
||||
|
||||
/// Number of live (non-evicted) observations.
|
||||
#[must_use]
|
||||
pub fn len(&self) -> usize {
|
||||
self.samples.len()
|
||||
}
|
||||
|
||||
/// Whether the window holds no live observations.
|
||||
#[must_use]
|
||||
pub fn is_empty(&self) -> bool {
|
||||
self.samples.is_empty()
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn weight_is_fresh_then_decays_then_drops() {
|
||||
let s = StalenessSchedule::new(4, 10, 0.5);
|
||||
assert_eq!(s.weight(0), 1.0);
|
||||
assert_eq!(s.weight(4), 1.0);
|
||||
let w5 = s.weight(5);
|
||||
assert!(w5 < 1.0 && w5 > 0.0); // decaying
|
||||
assert!(s.weight(9) < w5); // monotone decay
|
||||
assert_eq!(s.weight(11), 0.0); // dropped past k2
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn weighted_mean_favors_recent() {
|
||||
// Old samples = 1.0, recent samples = 0.0; the weighted mean must sit
|
||||
// well below the unweighted 0.5 because recent dominates.
|
||||
let mut w = StalenessWindow::new(StalenessSchedule::new(2, 32, 0.3), 64);
|
||||
for _ in 0..20 {
|
||||
w.push(1.0);
|
||||
}
|
||||
for _ in 0..20 {
|
||||
w.push(0.0);
|
||||
}
|
||||
let m = w.weighted_mean().unwrap();
|
||||
assert!(
|
||||
m < 0.25,
|
||||
"recent-weighted mean {m} should be near the recent 0.0"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn stale_observations_are_evicted() {
|
||||
let mut w = StalenessWindow::new(StalenessSchedule::new(2, 8, 0.5), 1000);
|
||||
for _ in 0..50 {
|
||||
w.push(1.0);
|
||||
}
|
||||
// Only observations within k2=8 ticks of the clock survive.
|
||||
assert!(w.len() <= 9, "expected <=9 live samples, got {}", w.len());
|
||||
assert!(w.weighted_mean().is_some());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn empty_window_has_no_mean() {
|
||||
let w = StalenessWindow::new(StalenessSchedule::default(), 8);
|
||||
assert!(w.weighted_mean().is_none());
|
||||
assert!(w.is_empty());
|
||||
}
|
||||
}
|
||||
297
crates/sona/src/darwin_guard.rs
Normal file
297
crates/sona/src/darwin_guard.rs
Normal file
|
|
@ -0,0 +1,297 @@
|
|||
//! Reward-hacking defenses for evolutionary harness/config search (ADR-271).
|
||||
//!
|
||||
//! Borrowed from Ornith-1.0's three-layer defense ("Self-Scaffolding LLMs for
|
||||
//! Agentic Coding", DeepReinforce 2026). When an evolutionary loop is allowed to
|
||||
//! evolve its own harness/config, candidates can "win" by gaming the fitness
|
||||
//! rather than improving — so the search must be screened:
|
||||
//!
|
||||
//! 1. **Immutable boundary** — the verifier (the fitness/eval) is frozen and
|
||||
//! lives outside what evolves; the genome can only change the *inner* policy.
|
||||
//! Modelled here by keeping [`screen`] a pure function of verifier output the
|
||||
//! candidate cannot fabricate.
|
||||
//! 2. **Deterministic monitor** — non-finite metrics, out-of-bounds genes, or a
|
||||
//! degenerate/collapsed "win" are flagged and the candidate is **excluded
|
||||
//! from the selection statistics** (Pareto front / advantage), NOT merely
|
||||
//! zero-scored. A zero-scored hack can still bias selection; an excluded one
|
||||
//! cannot. See [`best_accepted`].
|
||||
//! 3. **Frozen judge veto** — an [`IntentJudge`] (e.g. a frozen LLM) may VETO
|
||||
//! intent-level gaming inside the allowed surface, but never *sets* the
|
||||
//! reward — it is a veto on top of the verifier, not the reward itself.
|
||||
|
||||
/// Outcome of screening one candidate. `Rejected` candidates are dropped from the
|
||||
/// selection statistics entirely (the "exclude from advantage" rule).
|
||||
#[derive(Clone, Copy, Debug, PartialEq)]
|
||||
pub enum Verdict {
|
||||
/// Passed all layers; carries the verifier fitness.
|
||||
Accepted(f32),
|
||||
/// Rejected; excluded from Pareto/advantage with a reason.
|
||||
Rejected(Reject),
|
||||
}
|
||||
|
||||
/// Why a candidate was rejected (telemetry + auditability).
|
||||
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
|
||||
pub enum Reject {
|
||||
/// A metric or the fitness was NaN/Inf.
|
||||
NonFinite,
|
||||
/// A gene was outside its declared bounds.
|
||||
OutOfBounds,
|
||||
/// "Won" via a collapsed/trivial path (caller-defined degeneracy check).
|
||||
Degenerate,
|
||||
/// The frozen intent-judge vetoed it.
|
||||
JudgeVeto,
|
||||
}
|
||||
|
||||
/// Layer 3: a frozen judge that may only VETO a candidate, never set its reward.
|
||||
pub trait IntentJudge {
|
||||
/// Return `true` to veto (reject) the candidate.
|
||||
fn veto(&self, fitness: f32) -> bool;
|
||||
}
|
||||
|
||||
/// Deterministic-only screening (no judge).
|
||||
#[derive(Clone, Copy, Debug, Default)]
|
||||
pub struct NoJudge;
|
||||
impl IntentJudge for NoJudge {
|
||||
fn veto(&self, _fitness: f32) -> bool {
|
||||
false
|
||||
}
|
||||
}
|
||||
|
||||
/// The reward-hacking guard.
|
||||
#[derive(Clone, Copy, Debug)]
|
||||
pub struct Guard<J: IntentJudge = NoJudge> {
|
||||
judge: J,
|
||||
}
|
||||
|
||||
impl Guard<NoJudge> {
|
||||
/// Deterministic-monitor-only guard (layers 1–2).
|
||||
#[must_use]
|
||||
pub fn deterministic() -> Self {
|
||||
Self { judge: NoJudge }
|
||||
}
|
||||
}
|
||||
|
||||
impl<J: IntentJudge> Guard<J> {
|
||||
/// Guard with a layer-3 intent judge.
|
||||
pub fn with_judge(judge: J) -> Self {
|
||||
Self { judge }
|
||||
}
|
||||
|
||||
/// Screen one candidate. `fitness`/`finite_metrics` come from the IMMUTABLE
|
||||
/// verifier (the candidate cannot fabricate them); `in_bounds`/`degenerate`
|
||||
/// are caller-supplied deterministic checks over the genome + its metrics.
|
||||
pub fn screen(
|
||||
&self,
|
||||
fitness: f32,
|
||||
finite_metrics: bool,
|
||||
in_bounds: bool,
|
||||
degenerate: bool,
|
||||
) -> Verdict {
|
||||
if !finite_metrics || !fitness.is_finite() {
|
||||
return Verdict::Rejected(Reject::NonFinite);
|
||||
}
|
||||
if !in_bounds {
|
||||
return Verdict::Rejected(Reject::OutOfBounds);
|
||||
}
|
||||
if degenerate {
|
||||
return Verdict::Rejected(Reject::Degenerate);
|
||||
}
|
||||
if self.judge.veto(fitness) {
|
||||
return Verdict::Rejected(Reject::JudgeVeto);
|
||||
}
|
||||
Verdict::Accepted(fitness)
|
||||
}
|
||||
}
|
||||
|
||||
/// Best ACCEPTED candidate, EXCLUDING every rejected one from the comparison
|
||||
/// (the Ornith "exclude from advantage" rule). `None` if all were rejected.
|
||||
/// NaN-safe: rejected non-finite candidates never reach the comparator.
|
||||
#[must_use]
|
||||
pub fn best_accepted(verdicts: &[Verdict]) -> Option<(usize, f32)> {
|
||||
verdicts
|
||||
.iter()
|
||||
.enumerate()
|
||||
.filter_map(|(i, v)| match v {
|
||||
Verdict::Accepted(f) => Some((i, *f)),
|
||||
Verdict::Rejected(_) => None,
|
||||
})
|
||||
.max_by(|a, b| a.1.partial_cmp(&b.1).unwrap_or(std::cmp::Ordering::Equal))
|
||||
}
|
||||
|
||||
/// Rejection counts by reason: `[non_finite, out_of_bounds, degenerate, judge_veto]`.
|
||||
#[must_use]
|
||||
pub fn reject_summary(verdicts: &[Verdict]) -> [usize; 4] {
|
||||
let mut c = [0usize; 4];
|
||||
for v in verdicts {
|
||||
if let Verdict::Rejected(r) = v {
|
||||
c[match r {
|
||||
Reject::NonFinite => 0,
|
||||
Reject::OutOfBounds => 1,
|
||||
Reject::Degenerate => 2,
|
||||
Reject::JudgeVeto => 3,
|
||||
}] += 1;
|
||||
}
|
||||
}
|
||||
c
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Contamination guard (weight-eft / ADR-198 borrow). The training-data analog of
|
||||
// the reward-hacking monitor: training or selecting on instances that appear in
|
||||
// the eval holdout is *fake lift*. Enforce strict train/eval instance-ID
|
||||
// disjointness — and surface what was excluded, never silently.
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
use std::collections::HashSet;
|
||||
|
||||
/// Train IDs that illegally appear in the eval holdout (the contamination set).
|
||||
#[must_use]
|
||||
pub fn contamination<'a>(
|
||||
train_ids: impl IntoIterator<Item = &'a str>,
|
||||
eval_holdout: &[&str],
|
||||
) -> Vec<String> {
|
||||
let holdout: HashSet<&str> = eval_holdout.iter().copied().collect();
|
||||
let mut bad: Vec<String> = train_ids
|
||||
.into_iter()
|
||||
.filter(|id| holdout.contains(id))
|
||||
.map(str::to_string)
|
||||
.collect();
|
||||
bad.sort();
|
||||
bad.dedup();
|
||||
bad
|
||||
}
|
||||
|
||||
/// `assertTrainEvalDisjoint` analog: `Err(overlapping_ids)` if any training
|
||||
/// instance is in the eval holdout, else `Ok(())`. Callers should treat `Err` as
|
||||
/// fatal — a contaminated training set produces fake held-out lift.
|
||||
///
|
||||
/// # Errors
|
||||
/// Returns the sorted, de-duplicated overlapping instance IDs.
|
||||
pub fn assert_train_eval_disjoint(
|
||||
train_ids: &[&str],
|
||||
eval_holdout: &[&str],
|
||||
) -> Result<(), Vec<String>> {
|
||||
let bad = contamination(train_ids.iter().copied(), eval_holdout);
|
||||
if bad.is_empty() {
|
||||
Ok(())
|
||||
} else {
|
||||
Err(bad)
|
||||
}
|
||||
}
|
||||
|
||||
/// Exporter-style contamination filter: split `items` into
|
||||
/// `(kept, excluded_by_holdout)` by their instance id, so the training set is
|
||||
/// disjoint from the eval holdout by construction. Pair with the export report
|
||||
/// (`excluded.len()`), never drop silently.
|
||||
pub fn filter_holdout<T>(
|
||||
items: Vec<T>,
|
||||
id_of: impl Fn(&T) -> &str,
|
||||
eval_holdout: &[&str],
|
||||
) -> (Vec<T>, Vec<T>) {
|
||||
let holdout: HashSet<&str> = eval_holdout.iter().copied().collect();
|
||||
let mut kept = Vec::new();
|
||||
let mut excluded = Vec::new();
|
||||
for it in items {
|
||||
if holdout.contains(id_of(&it)) {
|
||||
excluded.push(it);
|
||||
} else {
|
||||
kept.push(it);
|
||||
}
|
||||
}
|
||||
(kept, excluded)
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn non_finite_is_excluded_not_zeroed() {
|
||||
let g = Guard::deterministic();
|
||||
// A NaN-producing candidate must be REJECTED (excluded), not scored 0 —
|
||||
// a 0 could still win if all real candidates score negative.
|
||||
assert_eq!(
|
||||
g.screen(f32::NAN, true, true, false),
|
||||
Verdict::Rejected(Reject::NonFinite)
|
||||
);
|
||||
assert_eq!(
|
||||
g.screen(1.0, false, true, false),
|
||||
Verdict::Rejected(Reject::NonFinite)
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn out_of_bounds_and_degenerate_rejected() {
|
||||
let g = Guard::deterministic();
|
||||
assert_eq!(
|
||||
g.screen(5.0, true, false, false),
|
||||
Verdict::Rejected(Reject::OutOfBounds)
|
||||
);
|
||||
assert_eq!(
|
||||
g.screen(5.0, true, true, true),
|
||||
Verdict::Rejected(Reject::Degenerate)
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn best_accepted_excludes_rejects_and_is_nan_safe() {
|
||||
// The hacked candidate (NonFinite) must NOT win even though its raw value
|
||||
// would sort highest; only accepted candidates are compared.
|
||||
let vs = [
|
||||
Verdict::Accepted(-0.5),
|
||||
Verdict::Rejected(Reject::NonFinite),
|
||||
Verdict::Accepted(-0.2),
|
||||
Verdict::Rejected(Reject::Degenerate),
|
||||
];
|
||||
assert_eq!(best_accepted(&vs), Some((2, -0.2)));
|
||||
assert_eq!(reject_summary(&vs), [1, 0, 1, 0]);
|
||||
// All rejected → no selection (caller must handle, not crash).
|
||||
assert_eq!(
|
||||
best_accepted(&[Verdict::Rejected(Reject::OutOfBounds)]),
|
||||
None
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn judge_vetoes_but_does_not_set_reward() {
|
||||
struct VetoHigh;
|
||||
impl IntentJudge for VetoHigh {
|
||||
fn veto(&self, fitness: f32) -> bool {
|
||||
fitness > 100.0 // an implausibly-good score smells like gaming
|
||||
}
|
||||
}
|
||||
let g = Guard::with_judge(VetoHigh);
|
||||
assert_eq!(
|
||||
g.screen(999.0, true, true, false),
|
||||
Verdict::Rejected(Reject::JudgeVeto)
|
||||
);
|
||||
assert_eq!(g.screen(1.0, true, true, false), Verdict::Accepted(1.0));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn disjoint_train_eval_ok_and_contamination_detected() {
|
||||
let eval = ["i-3", "i-9"];
|
||||
assert_eq!(assert_train_eval_disjoint(&["i-1", "i-2"], &eval), Ok(()));
|
||||
// Overlap is fatal and reports the contaminated ids (sorted, deduped).
|
||||
assert_eq!(
|
||||
assert_train_eval_disjoint(&["i-1", "i-9", "i-3", "i-9"], &eval),
|
||||
Err(vec!["i-3".to_string(), "i-9".to_string()])
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn filter_holdout_partitions_by_id() {
|
||||
let items = vec![("i-1", 10), ("i-3", 20), ("i-5", 30)];
|
||||
let (kept, excluded) = filter_holdout(items, |x| x.0, &["i-3"]);
|
||||
assert_eq!(
|
||||
kept.iter().map(|x| x.0).collect::<Vec<_>>(),
|
||||
vec!["i-1", "i-5"]
|
||||
);
|
||||
assert_eq!(
|
||||
excluded.iter().map(|x| x.0).collect::<Vec<_>>(),
|
||||
vec!["i-3"]
|
||||
);
|
||||
// The kept set is now disjoint from the holdout by construction.
|
||||
let kept_ids: Vec<&str> = kept.iter().map(|x| x.0).collect();
|
||||
assert!(assert_train_eval_disjoint(&kept_ids, &["i-3"]).is_ok());
|
||||
}
|
||||
}
|
||||
|
|
@ -45,6 +45,8 @@
|
|||
|
||||
#![allow(missing_docs)]
|
||||
|
||||
pub mod auto_tuner;
|
||||
pub mod darwin_guard;
|
||||
pub mod engine;
|
||||
pub mod ewc;
|
||||
pub mod loops;
|
||||
|
|
|
|||
67
docs/adr/ADR-271-metaharness-darwin-sona-self-improvement.md
Normal file
67
docs/adr/ADR-271-metaharness-darwin-sona-self-improvement.md
Normal file
|
|
@ -0,0 +1,67 @@
|
|||
# ADR-271: Metaharness-Darwin for SONA Self-Improvement — EWC Config Evolution, the weightAdapter Gene, and Ornith-1.0 Reward-Hacking Defenses
|
||||
|
||||
- **Status**: Proposed (all four components prototyped — PR #615)
|
||||
- **Date**: 2026-06-27
|
||||
- **Extends**: ADR-266 (metaharness-Darwin ANN optimization), ADR-269/270 (mragent graph-memory Darwin)
|
||||
- **External anchor**: Ornith-1.0 "Self-Scaffolding LLMs for Agentic Coding" (DeepReinforce, Jun 2026)
|
||||
|
||||
---
|
||||
|
||||
## Context
|
||||
|
||||
SONA (`crates/sona`) is a self-learning substrate: online LoRA adaptation + **EWC++** (`EwcPlusPlus`) to resist catastrophic forgetting, recording `QueryTrajectory` rewards. Its behaviour is governed by hand-tuned hyper-parameters (`EwcConfig`: lambda schedule, Fisher decay, task-boundary threshold; LoRA rank; MoE gate). These are **non-differentiable, workload-dependent** meta-parameters — gradients tune the weights, but nothing tunes the *config that governs how the weights are tuned*.
|
||||
|
||||
The metaharness line (ADR-266/269/270) established "freeze the model, evolve the harness" via **external evolutionary search**. This ADR applies that line to SONA's continual-learning layer and hardens it using the structure of **Ornith-1.0**, which does the same thing the *other* way (in-weights RL co-optimizing scaffold + solution).
|
||||
|
||||
## Decision
|
||||
|
||||
Apply metaharness-Darwin to SONA across four components. The **frozen model** is the EWC++/LoRA *algorithm*; the **evolved harness** is its config genome. Weights stay on the gradient path — Darwin only ever touches the meta-layer.
|
||||
|
||||
### 1. EWC++ config evolution (implemented — PR #615)
|
||||
Evolve the `EwcConfig` genome (GA + coordinate-descent polish) on a continual-learning benchmark with **no replay** and **self-detected task boundaries**, train/test split over task-sequence seeds. Measured (held-out): **35% lower final loss, 98.6% less forgetting** than `EwcConfig::default()` (the crate's hand-tuned "OPTIMIZED" values) — a strict Pareto win that generalizes to unseen sequences. (`examples/darwin_ewc.rs`)
|
||||
|
||||
### 2. The `weightAdapter` gene (implemented — PR #615)
|
||||
Expose a fine-tuned adapter delta (a LoRA) as a gene `(which_adapter, alpha)` so evolutionary selection decides *whether/how much* to apply it (`w_eff = w_base + alpha·Δw`) rather than assuming new weights are better. **Key finding** (`examples/darwin_weightadapter.rs`): "selection prunes overfit adapters" holds **only under per-domain (no-regression Pareto) evaluation**. A volume-weighted aggregate fitness is fooled by an adapter whose in-dist gain (where the eval pool concentrates) hides an out-dist regression. → **Score every adapter per-repository.**
|
||||
|
||||
### 3. The Autonomous Data Engine (realized upstream as `@metaharness/weight-eft`, ADR-198)
|
||||
Darwin's archive is **execution-verified** preference data — the label is a *passing test suite*, not a noisy human judgment (higher-signal than RLHF). `@metaharness/weight-eft` realizes this for the agentic-coding cost-cascade (SFT/DPO distillation of gold SWE-bench trajectories into a cheap-tier LoRA via ruvllm/MicroLoRA, to escalate to a frontier model less often). Adopt its recipe — it gets the RLHF-correctness right:
|
||||
|
||||
- **SFT** distills **all** gold-resolved trajectories (cheap-own *and* frontier-escalation): max-likelihood is off-policy-stable, so a frontier success on an issue the cheap model couldn't solve is safe to distill.
|
||||
- **DPO is on-policy only**: `chosen`/`rejected` are the **same model on the same instance** (cheap-vs-cheap, BoN-derived). A frontier-chosen-vs-cheap-rejected pair is off-policy/unstable → route it to SFT, not DPO. (Supersedes the earlier "plausible-but-failed negatives" sketch with the correct on/off-policy split.)
|
||||
- **Contamination guard**: strict **train/eval instance-ID disjointness** — training on eval instances is fake lift. Implemented SONA-side in `darwin_guard` (`contamination` / `assert_train_eval_disjoint` / `filter_holdout`) as the analog of weight-eft's `assertTrainEvalDisjoint`.
|
||||
- Portable export (OpenAI-chat JSONL with `tool_calls` preserved; TRL DPO); the trained adapter plugs back in as the `weightAdapter` gene (§2). The ruvllm/MicroLoRA seam is the ruvector integration point.
|
||||
|
||||
### 4. Ornith-1.0 borrows (method, not model)
|
||||
Ornith bakes scaffold-evolution into weights via RL; we keep it external (cheaper, model-agnostic, no training). We borrow its *structure*:
|
||||
|
||||
- **3-layer reward-hacking defense + contamination guard** (the `darwin-guard` module): (i) **immutable outer boundary** — the verifier/eval is frozen and outside what evolves; (ii) **deterministic monitor** — gated variants (new imports/network/shell/env, reading withheld paths, touching the verifier) are **excluded from the advantage/Pareto computation**, not merely zero-scored, so they cannot bias selection; (iii) **frozen LLM judge as a veto** (local GPU `qwen`) on intent-level Goodharting inside the allowed surface — a veto on top of the verifier, never the primary reward; plus (iv) **train/eval contamination guard** (weight-eft / ADR-198 borrow): `assert_train_eval_disjoint` fails on any train∩eval instance-ID overlap — training/selecting on eval instances is fake lift.
|
||||
- **Per-task-category specialization**: evolve a router `task-class → genome` instead of one global genome (Ornith's main empirical result is per-category strategies emerging).
|
||||
- **Two-stage reward credit**: credit the *mutation/scaffold-proposal* that produced a winning genome, not just the outcome — turning the random `mutate()` into a learned write-layer (and the `(proposal → outcome)` pairs are themselves data-engine preference pairs).
|
||||
- **Staleness-weighted replay** `w(d_t)` (1 if fresh → exp-decay → drop past threshold) for the online auto-tuner over SONA's live trajectory stream; maps onto `fisher_ema_decay` and is itself evolvable.
|
||||
|
||||
## Consequences
|
||||
|
||||
**Positive**: out-tunes hand-tuning on held-out continual learning (measured); model-agnostic and training-free (vs Ornith's GPU-scale RL); the reward-hacking defenses make the loop rigorous and Goodhart-resistant; the same Darwin genome co-optimizes "adopt this fine-tune?" with "how hard to protect old knowledge?".
|
||||
|
||||
**Negative / risks**: a beyond-SOTA number is only as real as the benchmark — the immutable-verifier boundary (borrow #4-i) is what keeps it honest; meta-optimization cost scales with benchmark realism (real nets ⇒ GPU + parallelism); generalization across *workload distributions* (not just task sequences) likely needs the per-category router (#4-ii), not one frozen genome.
|
||||
|
||||
## Relationship to Ornith-1.0
|
||||
| | Ornith-1.0 | This ADR |
|
||||
|---|---|---|
|
||||
| Harness optimization | in-weights RL (gradients), two-stage | external evolutionary (GA/Pareto) |
|
||||
| Cost | frontier RL training (9B–397B) | training-free, any frozen model |
|
||||
| Reward-hack defense | immutable boundary + monitor + judge | **borrowed verbatim** (darwin-guard) |
|
||||
| Specialization | per-task-category (emergent) | per-task-category router (borrowed) |
|
||||
|
||||
Complementary, not competing: external-Darwin is the no-training counterpart to Ornith's in-weights approach.
|
||||
|
||||
## Implementation status
|
||||
- ✅ EWC config evolution + weightAdapter gene (PR #615, `feat/sona-darwin-ewc-evolve`).
|
||||
- ✅ darwin-guard reward-hacking + contamination module (`crates/sona/src/darwin_guard.rs`, 6 tests; reward-hacking screen wired into `darwin_ewc`; `assert_train_eval_disjoint`/`filter_holdout` = the weight-eft contamination guard).
|
||||
- ✅ per-task-category router (`examples/darwin_router.rs`): beats the single best global config on held-out (~2%), with the **data-efficiency caveat** — the gain *reverses* when per-class data is scarce (a specialized config overfits while the pooled global generalizes), so routing needs enough per-category samples (Ornith's regime).
|
||||
- ✅ online auto-tuner with staleness-weighted replay `w(d_t)` (`crates/sona/src/auto_tuner.rs` — `StalenessSchedule`/`StalenessWindow`, 4 tests; `examples/darwin_autotuner.rs` — a (1+1)-ES that adapts a deployed config to workload drift, beating the static config ~3% post-drift). Modest margin on synthetic regimes; the durable win is the reusable staleness machinery + the online-adaptation principle (a fixed offline-tuned config goes stale under drift).
|
||||
|
||||
## References
|
||||
- Ornith-1.0: "Self-Scaffolding LLMs for Agentic Coding", DeepReinforce, 2026-06.
|
||||
- `@metaharness/weight-eft` (npm) — evolutionary fine-tuning / autonomous data engine (SFT + on-policy DPO → cheap-tier LoRA), `agent-harness-generator` ADR-198. The production realization of §3 + §2; this ADR borrows its on-policy-DPO recipe and contamination-disjointness guard.
|
||||
- ADR-266 metaharness-Darwin; ADR-269/270 mragent; PR #615.
|
||||
Loading…
Add table
Add a link
Reference in a new issue