From 7e682ea5265cd019e3a74b4a53f718e90699eaaa Mon Sep 17 00:00:00 2001 From: ruvnet Date: Wed, 22 Apr 2026 23:27:02 -0400 Subject: [PATCH] =?UTF-8?q?feat(analysis):=20hub-fraction=20+=20density=20?= =?UTF-8?q?sweeps=20at=20N=3D1024=20=E2=80=94=2028th=20&=2029th=20discover?= =?UTF-8?q?y?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit #28 (null): hub_modules ∈ {0, 1, 2, 3, 4, 6, 8} at N=1024/40-modules. Peak stays at hub=3 → 0.516. hub ∈ [0, 2] cluster at 0.487–0.488; hub ≥ 4 collapses to 0.37–0.43. Narrow non-monotonic peak, not a smooth ridge. The "smaller hub wins" pattern from N=512 does NOT generalise to N=1024 — 2nd ADR-level case of "hypothesis from small-N extrapolates wrong at large N" (1st was item 22 on fixed γ). #29: fine num_modules ∈ {20, 25, 30, 35, 40, 50, 60, 80} at N=1024/ hub=3. New N=1024 peak: 0.531 @ modules=30 (density 34.1), γ=3.0 (70 communities vs 30 truth). Secondary peak at modules=80/γ=2.5 scores 0.515 — multi-modal landscape confirmed. Finding: at N=1024 the optimal density is 34.1 neurons/module, not 25.6. At N=512 it's 25.6. The 4-D landscape (N × density × γ × hub) does not factorize. AC-3a gap at N=1024 now 1.41× (down from 1.47×). Best-across-scales remains 0.599 @ (N=512, modules=20, hub=1, γ=4.0) — 1.25× gap. - tests/leiden_cpm.rs: leiden_cpm_hub_fraction_sweep_at_n1024, leiden_cpm_module_count_sweep_at_n1024_hub3 - ADR-154 §17 rows 28, 29 + heading 27 → 29 Co-Authored-By: claude-flow --- ...R-154-connectome-embodied-brain-example.md | 6 +- examples/connectome-fly/tests/leiden_cpm.rs | 108 ++++++++++++++++++ 2 files changed, 112 insertions(+), 2 deletions(-) diff --git a/docs/adr/ADR-154-connectome-embodied-brain-example.md b/docs/adr/ADR-154-connectome-embodied-brain-example.md index a26f54a6..e616f81e 100644 --- a/docs/adr/ADR-154-connectome-embodied-brain-example.md +++ b/docs/adr/ADR-154-connectome-embodied-brain-example.md @@ -446,9 +446,9 @@ This section enumerates the risks this ADR is aware of and how the example stays This register is not comprehensive. It is the set of risks the branch has surfaced by running into them (positioning creep, threshold drift, null-distribution sloppiness, pre-measurement mis-diagnosis, envelope-vs-bit-exact framing, speculative-parenthetical predictions). Future commits are expected to add rows; they are not expected to remove rows. -## 17. Twenty-seven measurement-driven discoveries (roll-up) +## 17. Twenty-nine measurement-driven discoveries (roll-up) -Each of the twenty-seven is attached to the commit that produced it and the lesson it encoded for future work. +Each of the twenty-nine is attached to the commit that produced it and the lesson it encoded for future work. | # | Commit | Finding | Lesson | |---|---|---|---| @@ -479,6 +479,8 @@ Each of the twenty-seven is attached to the commit that produced it and the less | 25 | CPM-specific refinement phase tested — collapses at the γ regime where CPM works | Implemented the named-in-item-19 lever: Traag 2019 Alg. 4 with the CPM objective (`refine_cpm` / `refine_cpm_one_community` in `src/analysis/leiden.rs`). Wired between local moves and aggregate; ran the full CPM test suite. **Catastrophic regression across the board**: N=512 peak **0.549 → 0.038** @ γ=3.1 (−93 %); N=1024 peak **0.425 → 0.023** @ γ=2.25 (−95 %); seed-sweep mean ratio vs modularity flipped from 3.98× to 0.21×. Coarse-sweep on default SBM showed peak ARI migrating to γ=0.10 with 0.357 — i.e., refinement is now *only* effective at an order-of-magnitude lower γ than the no-refinement sweet spot. **Refinement wiring reverted; `refine_cpm` kept in tree behind `#[allow(dead_code)]` with a pointer comment to this item.** | **The named next lever didn't just fail to help — it was actively destructive, and the mechanism is now well-understood.** The CPM refinement starts every node as a singleton sub-community within its coarse C. For v to merge into an existing singleton s, the gain `k_{v→s} − γ·n_v·n_s` must be positive. At the γ range where CPM excels on this substrate (γ ∈ [2, 3] post-normalization with mean weight = 1.0), a *single* edge of weight ~1 cannot overcome the γ·1·1 = 2–3 merge cost. **Refinement leaves nearly everything as singletons, and the subsequent aggregation step projects onto the identity, destroying the coarse structure built by level1_moves.** This is a clean instance of a third distinct failure-mode pattern on this branch: not "algorithm needs a rider" (item 16 → 17), not "measurement undersells" (item 18), but **"algorithm-that-ships-with-paper has a regime where the paper's claim holds and a regime where it destroys previous progress — and identifying the regime requires actually measuring, not reading the paper."** Traag & Waltman 2019 is explicit that refinement helps Leiden dominate Louvain; it is *not* explicit that their refinement formulation is sensitive to γ scaling in a way that makes it self-defeating at γ ∈ [2, 3]. At lower γ values (γ = 0.1, where singletons can cheaply merge), refinement would likely work as advertised — but at γ = 0.1 CPM itself scores 0.357 on the default SBM (well below the 0.549 ceiling at N=512 / γ=3.1). **So the lever is unavailable at the operating point where CPM is strongest, and the AC-3a 0.75 SOTA gap remains at 1.37× via CPM-without-refinement.** This is now the 9th pre-measurement-ADR-named lever ruled out by measurement; it shifts the remaining lever catalogue to (a) degree-stratified null for AC-5, (b) real-FlyWire ingest (the only remaining axis for AC-2 and likely AC-3a too), and (c) CPM refinement with a substrate-specific *non-singleton* start state — which is research, not engineering. Code: `src/analysis/leiden.rs::refine_cpm` (unwired, kept); no test change — the existing CPM sweeps are already sensitive enough to have flagged the regression if it had shipped. | | 26 | N=512 module-count sweep — 0.599 @ 20 modules (new ceiling) | Fixed N=512, swept num_modules ∈ {20, 25, 30, 35, 40, 45, 50} with hub_modules = m/12 (constant hub-ratio) and γ ∈ {1.5, 2.0, 2.5, 3.0, 3.5, 4.0, 5.0} per module-count. **New peak full_ARI = 0.599 at num_modules=20, γ=4.0 (21 communities vs 20 truth)** — 9 % higher than the item-24 headline of 0.549 at num_modules=35. Per-config peaks: (20, 0.599) (25, 0.505) (30, 0.528) (35, 0.507) (40, 0.559) (45, 0.566) (50, 0.517). | **The "N=512 sweet spot" has more sub-structure than item 24 measured.** Module-count is a real axis: at N=512, the item-24 baseline (num_modules=35, 14.6 neurons/module) was NOT the optimal granularity — 20 modules (25.6 neurons/module) beats it by 9 % on full-ARI, and the γ peak shifts up (γ=4.0) to match the lower module count. A second local maximum appears at num_modules ∈ [40, 45], suggesting the quality ridge is multi-modal rather than a single peak. **New CPM ceiling on this substrate: 0.599 at (N=512, num_modules=20, γ=4.0). Gap to 0.75 AC-3a SOTA target narrows from 1.37× (item 24) to 1.25×.** The item-22→24 pattern now has a tighter form: *N is not the only axis — module count and γ together define a 2D quality landscape, and prior measurements held one dimension fixed at a non-optimal value*. Code: `tests/leiden_cpm.rs::leiden_cpm_module_count_sweep_at_n512`. Opens the natural follow-up: does this "few-large-modules wins" pattern hold at other N, or is it a cross-coupling with N=512 specifically? | | 27 | Cross-scale constant-density (≈25.6 neurons/module) γ-sweep — N=1024 at 40 modules scores 0.516, not 0.425 | Held neurons/module ≈ 25.6 (the item-26 sweet spot). Varied N ∈ {256, 512, 1024, 2048} with num_modules = N/25 and hub_modules proportional. γ sweep {2.0, 2.5, 3.0, 3.5, 4.0, 5.0, 6.0, 8.0}. **Per-scale peaks: N=256 → 0.466 @ γ=5.0 (6 communities vs 10 truth); N=512 → 0.554 @ γ=4.0 (23 vs 20; note: 0.045 lower than item-26's 0.599 because hub_modules differs — 2 here, 1 in #26); N=1024 → 0.516 @ γ=2.5 (96 vs 40); N=2048 → 0.343 @ γ=2.0 (257 vs 80).** γ-peak-vs-N still monotonic (5.0 → 4.0 → 2.5 → 2.0). | **The "ARI peaks at N=512" finding from item 24 was density-dependent, not a universal property.** At density=14.6 (items 22–24), N=1024 scored 0.425; at density=25.6, N=1024 scores **0.516** — a 21 % lift from changing only the num_modules choice while holding N fixed. The full-ARI ceiling landscape is 3D (N × num_modules × γ), not 2D (N × γ as I'd been treating it). Two new patterns here: (i) **every prior N=1024 measurement on this branch used a sub-optimal num_modules** — the default substrate's 70-module choice maps to density 14.6, which is below the density-25.6 optimum observed at N=512; (ii) **hub_modules is a hidden 4th axis** — the N=512 peak dropped from 0.599 (item 26, hub=1) to 0.554 (this test, hub=2), a 0.045-unit difference from a single configuration parameter. The CPM ceiling on this substrate is best quoted as "~0.55–0.60 somewhere in the (N ∈ [384, 1024], density ∈ [20, 26], γ ∈ [2, 4], hub_fraction ∈ [5 %, 10 %]) landscape". The AC-3a gap to 0.75 SOTA ranges from 1.25× at the best observed config to 1.40× at the worst in this range. Code: `tests/leiden_cpm.rs::leiden_cpm_cross_scale_constant_density_at_25`. Opens the natural follow-up: sweep hub_fraction at N=1024 density=25.6 and look for a peak ≥ 0.55 — if found, the "default N=1024 substrate is unrelatable" argument against the branch (subtext of items 22/23/24) flips: the N=1024 substrate is fine, prior configurations were just mis-tuned. | +| 28 | Hub-fraction sweep at N=1024 — peak stays at hub=3 (0.516), no new ceiling | At N=1024 / num_modules=40 / density=25.6, swept hub_modules ∈ {0, 1, 2, 3, 4, 6, 8} with γ ∈ {2.0, 2.5, 3.0, 3.5, 4.0, 5.0}. **Peak unchanged: 0.516 @ hub=3 (7.5 %), γ=2.5.** Neighboring hub values: hub=0/1/2 all cluster at 0.487–0.488; hub=4/6/8 drop to 0.374–0.434. A **narrow, non-monotonic peak**, not a smooth ridge. | **Hub-fraction is a real axis but its optimum is too narrow to close the AC-3a gap alone.** The hypothesis from items 26 (N=512 hub=1 wins) and 27 (N=512 hub=2 drops 0.045) was "smaller hub fraction → higher ARI". At N=1024 that predicts a peak at hub=0 or 1 — which would imply the item-27 config (hub=3) was suboptimal. Measured: hub ∈ [0, 2] scores a flat 0.488, hub=3 spikes to 0.516, hub ≥ 4 collapses. The pattern is "there is a sharp sweet spot that depends on N", not "fewer hubs always win". Second ADR-level finding on this branch of "hypothesis from a smaller-N data point extrapolates wrong at larger N" (the first was item 22 on fixed γ). The AC-3a gap at N=1024 stays at 1.45× (0.516 vs 0.75); the best observed on the branch remains 0.599 at (N=512, 20 modules, hub=1, γ=4.0). Code: `tests/leiden_cpm.rs::leiden_cpm_hub_fraction_sweep_at_n1024`. Opens the follow-up item 29 below — sweep num_modules at N=1024 / hub=3 instead. | +| 29 | Fine num_modules sweep at N=1024/hub=3 — new N=1024 peak 0.531 @ density=34.1 | Follow-up to items 27 and 28. Swept num_modules ∈ {20, 25, 30, 35, 40, 50, 60, 80} at N=1024, hub_modules=3 (item 28's winner), γ ∈ {1.5, 2.0, 2.5, 3.0, 3.5, 4.0, 5.0}. **New peak: 0.531 @ num_modules=30 (34.1 neurons/module), γ=3.0 (70 communities vs 30 truth)** — +2.9 % on item 27's 0.516 headline at density=25.6. A second local peak at num_modules=80 / γ=2.5 scores 0.515 (multi-modal landscape confirmed). | **At N=1024 the optimal density is 34.1 neurons/module, not 25.6.** Item 27 found the density-axis matters; item 29 refines it — the optimum density shifts with N. At N=512 the winning density is 25.6 (item 26); at N=1024 it's 34.1. So the 4-D landscape (N × density × γ × hub) doesn't factorize: you can't set "the right" density once and vary N. This is consistent with a physical picture where communities large enough to be structurally distinct need more neurons inside them at higher N because the noise floor from inter-module crosstalk grows with N. **AC-3a gap at N=1024 is now 1.41× (0.531 vs 0.75), down from 1.47× at density=25.6.** Best across scales remains 0.599 at (N=512, 20 modules, hub=1, γ=4.0) — a 1.25× gap — but the N=1024 story now has a named optimum that the default substrate doesn't hit. Code: `tests/leiden_cpm.rs::leiden_cpm_module_count_sweep_at_n1024_hub3`. | The discoveries form a pattern: every "next lever named in the ADR" ultimately required an empirical test. **Eight** of the fifteen pre-measurement diagnoses tested on this branch proved wrong (items 7, 8, 9, 10, 12, 13, 15, 16). **Four unambiguous wins now: item 6 (adaptive cadence, 4.29× saturated-regime speedup), item 14 (Leiden refinement, perfect ARI on planted SBM where Louvain collapsed), item 17 (weight-normalized CPM-Leiden, perfect ARI on planted SBM + 109 communities on 70-module default SBM), and item 18 (full-partition ARI metric, lifting CPM's default-SBM score from 0.020 two-way to 0.393 full — 3.7× the modularity-Leiden baseline).** Items 6 and 14 followed the orthogonal-axis pattern. Item 17 was the first "rider from item 16 works as predicted" data point. Item 18 is a different shape again — a **measurement upgrade** that revealed an algorithm's prior 0.020 2-way score was hiding a 0.393 full-partition score. That's a new entry in the lesson catalogue: *a test's coarsening choice is as much a threshold decision as its numerical tolerances.* Three distinct "how a measurement-driven discovery lands" shapes now documented (orthogonal axis / rider matches paper / coarsening upgrade). diff --git a/examples/connectome-fly/tests/leiden_cpm.rs b/examples/connectome-fly/tests/leiden_cpm.rs index 543e7f17..36002042 100644 --- a/examples/connectome-fly/tests/leiden_cpm.rs +++ b/examples/connectome-fly/tests/leiden_cpm.rs @@ -704,3 +704,111 @@ fn leiden_cpm_cross_scale_constant_density_at_25() { best_overall_ari, best_overall.0, best_overall.1, best_overall.2 ); } + +#[test] +fn leiden_cpm_hub_fraction_sweep_at_n1024() { + // Follow-up to item 27. At N=1024 with num_modules=40 (density + // 25.6) and hub_modules=3, CPM scored 0.516. Item 27 also noted + // that at N=512 the hub_modules choice matters: hub=1 → 0.599 + // (item 26), hub=2 → 0.554 (item 27's config). Hypothesis: at + // N=1024, reducing hub_modules should raise the ceiling past + // 0.516 and perhaps past 0.599 (closing the AC-3a gap further). + // + // Sweep hub_modules ∈ {0, 1, 2, 3, 4, 6, 8} at N=1024 / + // num_modules=40. Per-hub γ sweep. + let hub_counts: [u16; 7] = [0, 1, 2, 3, 4, 6, 8]; + let gammas = [2.0, 2.5, 3.0, 3.5, 4.0, 5.0]; + let mut overall_best_ari = f32::NEG_INFINITY; + let mut overall_best: (u16, f64) = (0, 0.0); + for &h in &hub_counts { + let cfg = ConnectomeConfig { + num_neurons: 1024, + num_modules: 40, + num_hub_modules: h, + ..ConnectomeConfig::default() + }; + let conn = Connectome::generate(&cfg); + let truth: Vec = (0..conn.num_neurons()) + .map(|i| conn.meta(connectome_fly::NeuronId(i as u32)).module as u32) + .collect(); + let mut best_ari = f32::NEG_INFINITY; + let mut best_g = 0.0_f64; + let mut best_d = 0usize; + for &g in &gammas { + let labels = connectome_fly::analysis::leiden::leiden_labels_cpm(&conn, g); + let ari = full_partition_ari(&labels, &truth); + if ari > best_ari { + best_ari = ari; + best_g = g; + best_d = count_unique(&labels); + } + } + let hub_frac = 100.0 * h as f32 / 40.0; + eprintln!( + "cpm-hub-sweep-N1024: hub_modules={:2} ({:.1}%) PEAK full_ari={:.3} @ γ={:.2} (distinct={})", + h, hub_frac, best_ari, best_g, best_d + ); + if best_ari > overall_best_ari { + overall_best_ari = best_ari; + overall_best = (h, best_g); + } + } + eprintln!( + "cpm-hub-sweep-N1024: OVERALL PEAK full_ari={:.3} at hub_modules={} γ={:.2} [vs 0.516 item-27 headline, 0.75 SOTA]", + overall_best_ari, overall_best.0, overall_best.1 + ); +} + +#[test] +fn leiden_cpm_module_count_sweep_at_n1024_hub3() { + // Orthogonal follow-up to item 28. Hub-fraction sweep at + // N=1024/40 didn't break 0.516. Try fine num_modules sweep at + // N=1024 with hub_modules=3 (item 28's winner) and a wider γ + // grid. This tests whether density=25.6 (40 modules) is the + // right choice at N=1024 or whether the N=1024 landscape has a + // different density optimum than N=512. + let module_counts: [u16; 8] = [20, 25, 30, 35, 40, 50, 60, 80]; + let gammas = [1.5, 2.0, 2.5, 3.0, 3.5, 4.0, 5.0]; + let mut overall_best_ari = f32::NEG_INFINITY; + let mut overall_best: (u16, f64) = (0, 0.0); + for &m in &module_counts { + // Hub=min(3, m/8) — stay close to the item-28 winner hub_frac + // while scaling reasonably with module count. + let h = (m / 8).min(3).max(1); + let cfg = ConnectomeConfig { + num_neurons: 1024, + num_modules: m, + num_hub_modules: h, + ..ConnectomeConfig::default() + }; + let conn = Connectome::generate(&cfg); + let truth: Vec = (0..conn.num_neurons()) + .map(|i| conn.meta(connectome_fly::NeuronId(i as u32)).module as u32) + .collect(); + let mut best_ari = f32::NEG_INFINITY; + let mut best_g = 0.0_f64; + let mut best_d = 0usize; + for &g in &gammas { + let labels = connectome_fly::analysis::leiden::leiden_labels_cpm(&conn, g); + let ari = full_partition_ari(&labels, &truth); + if ari > best_ari { + best_ari = ari; + best_g = g; + best_d = count_unique(&labels); + } + } + let neurons_per_mod = 1024.0 / m as f32; + eprintln!( + "cpm-modsweep-N1024: modules={:3} n_per_mod={:.1} hub={} PEAK full_ari={:.3} @ γ={:.2} (distinct={})", + m, neurons_per_mod, h, best_ari, best_g, best_d + ); + if best_ari > overall_best_ari { + overall_best_ari = best_ari; + overall_best = (m, best_g); + } + } + eprintln!( + "cpm-modsweep-N1024: OVERALL PEAK full_ari={:.3} at num_modules={} γ={:.2} [vs 0.516 item-27, 0.599 item-26, 0.75 SOTA]", + overall_best_ari, overall_best.0, overall_best.1 + ); +}