vrr/WFGY
2
0
Fork 0
mirror of https://github.com/onestardao/WFGY.git synced 2026-07-09 15:58:34 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 4
WFGY/Polaris/experiments/downloads/PP01.ipynb
PSBigBig + MiniPS c5652e7302
Add files via upload
2026-05-06 17:23:22 +08:00

2158 lines
124 KiB
Text
Raw Permalink Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

{
"cells": [
{
"cell_type": "markdown",
"id": "c5b4ab1e",
"metadata": {},
"source": [
"# PP01 — OneShotKiller 320 Main Cost Battlefield\n",
"\n",
"**Belongs to:** WFGY 5.0 Polaris Protocol \n",
"**Repository:** https://github.com/onestardao/WFGY \n",
"**Polaris Protocol path:** https://github.com/onestardao/WFGY/tree/main/Polaris \n",
"**Experiment page:** https://github.com/onestardao/WFGY/blob/main/Polaris/experiments/README.md\n",
"\n",
"## Experiment spirit\n",
"\n",
"Main cost battlefield. This is where Polaris Protocol asks whether task topology can beat raw-context drag under scoped gates.\n",
"\n",
"## What this Colab tests\n",
"\n",
"Tests whether topology-first task structure can reduce repeated long-context token workload while preserving inspectable quality and boundary behavior.\n",
"\n",
"## Published result summary\n",
"\n",
"- 320 cases; 6 arms; 1920 expected outputs and 1920 actual outputs.\n",
"- Structured C arm total token ratio vs A: 0.1656.\n",
"- Structured C arm input token ratio vs A: 0.1242.\n",
"- Overclaim count: 0; critical wrong-source count: 0; family collapse count: 0.\n",
"\n",
"## Expected usage / token-cost note\n",
"\n",
"Full run can be expensive. Previous 320-case runs were roughly around 1.2M–1.4M total tokens; actual usage depends on selected model, retry behavior, and output length.\n",
"\n",
"## Scientific boundary\n",
"\n",
"Scoped cost / quality / boundary evidence only. Not universal token-saving proof, not invoice proof, not third-party certification.\n",
"\n",
"This Colab is provided for **reproduction, inspection, and adaptation**. You do not have to rerun it to read the published result summary. If you rerun it, model behavior, token usage, cost, and output details may vary.\n",
"\n",
"---\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "d2f70fef",
"metadata": {},
"outputs": [],
"source": [
"# Polaris Protocol experiment identity\n",
"POLARIS_PROTOCOL_REPO = \"https://github.com/onestardao/WFGY\"\n",
"POLARIS_EXPERIMENT_ID = \"PP01\"\n",
"POLARIS_EXPERIMENT_TITLE = \"PP01 — OneShotKiller 320 Main Cost Battlefield\"\n",
"POLARIS_EXPERIMENT_ROLE = \"OneShotKiller 320 Main Cost Battlefield\"\n",
"POLARIS_CLAIM_BOUNDARY = \"Scoped cost / quality / boundary evidence only. Not universal token-saving proof, not invoice proof, not third-party certification.\"\n",
"print(f\"{POLARIS_EXPERIMENT_ID} — {POLARIS_EXPERIMENT_TITLE}\")\n",
"print(f\"Repo: {POLARIS_PROTOCOL_REPO}\")\n"
]
},
{
"cell_type": "markdown",
"id": "0428dced",
"metadata": {},
"source": [
"# WFGY 5.0 Polaris Protocol\n",
"\n",
"## PP01 · One Shot Killer MVP · 320 Case Public Reproducibility Run · v8 Progress and Resume\n",
"\n",
"Repo: https://github.com/onestardao/WFGY \n",
"License: MIT License\n",
"\n",
"---\n",
"\n",
"## TLDR for engineers\n",
"\n",
"This notebook tests whether long context reasoning can be compiled into a compact topology packet.\n",
"\n",
"Instead of sending the full context to a large model every time, WFGY extracts the decision critical structure and sends that compact structure to a smaller model.\n",
"\n",
"The benchmark checks whether this cheaper path preserves the same decision behavior.\n",
"\n",
"It does this with a six arm audit:\n",
"\n",
"| Arm | Path | Purpose |\n",
"|---|---|---|\n",
"| A | full context baseline | expensive reference path |\n",
"| B | teacher topology reference | structured teacher path |\n",
"| C | compact small model path | main low cost candidate |\n",
"| D | corrupted topology path | failure signature test |\n",
"| E | core ablation path | causal structure test |\n",
"| F | missing evidence / condition flip path | stop and flip consistency test |\n",
"\n",
"The expected engineering signal is:\n",
"\n",
"```text\n",
"high C retention\n",
"low C token ratio\n",
"strong D/E/F causal checks\n",
"zero wrong source critical errors\n",
"zero shadow overclaims\n",
"zero family collapse\n",
"```\n",
"\n",
"If this run passes, it supports a scoped but important result:\n",
"\n",
"```text\n",
"Compact topology packets can preserve decision behavior for PP01 topology conservation tasks at much lower token cost.\n",
"```\n",
"\n",
"---\n",
"\n",
"## What this experiment is\n",
"\n",
"This notebook is the first main public reproducibility experiment for **WFGY 5.0 Polaris Protocol**.\n",
"\n",
"The core question is simple:\n",
"\n",
"```text\n",
"Can a long context reasoning task be compiled into a compact task topology,\n",
"then solved by a smaller model at much lower token cost\n",
"without losing the decision structure?\n",
"```\n",
"\n",
"This is not a prompt trick.\n",
"\n",
"This is a compiler style experiment.\n",
"\n",
"The full context is not merely summarized. It is converted into a structured task object containing the minimum decision critical fields.\n",
"\n",
"For each task, the benchmark tracks:\n",
"\n",
"1. who controls the decision\n",
"2. which evidence is active\n",
"3. what boundary applies\n",
"4. what current condition is true\n",
"5. how far the final claim is allowed to go\n",
"\n",
"In this experiment, that compact structure is called a **topology packet**.\n",
"\n",
"The important test is not whether a model can write a plausible answer. The important test is whether the compact packet preserves the same decision behavior as the full long context.\n",
"\n",
"---\n",
"\n",
"## Why engineers should care\n",
"\n",
"Most long context pipelines pay for the same expensive operation again and again:\n",
"\n",
"```text\n",
"read a large noisy context\n",
"find the relevant authority\n",
"identify active evidence\n",
"ignore stale or non controlling sources\n",
"respect the boundary\n",
"produce a scoped answer\n",
"```\n",
"\n",
"WFGY tests a different architecture:\n",
"\n",
"```text\n",
"long context\n",
"→ task topology extraction\n",
"→ compact topology packet\n",
"→ small model reasoning\n",
"→ verifier and audit\n",
"```\n",
"\n",
"The cost reduction comes from moving work out of repeated full context inference.\n",
"\n",
"Instead of asking a large model to repeatedly re read the whole context, the system compiles the task into a smaller structure and sends that structure to a cheaper model.\n",
"\n",
"The experiment compares that compact path against a full context baseline.\n",
"\n",
"---\n",
"\n",
"## What this notebook actually runs\n",
"\n",
"This is a deterministic 320 case benchmark.\n",
"\n",
"Each case is expanded into six different model inputs.\n",
"\n",
"| Arm | Input | Purpose |\n",
"|---|---|---|\n",
"| A | large model with full long context | cost and full context baseline |\n",
"| B | large model with teacher topology packet | topology reference |\n",
"| C | small model with compact topology packet | main low cost candidate |\n",
"| D | small model with corrupted topology packet | failure signature test |\n",
"| E | small model with core fields removed | causal ablation test |\n",
"| F | small model with missing evidence or changed condition | stop and flip consistency test |\n",
"\n",
"Total outputs:\n",
"\n",
"```text\n",
"320 cases × 6 arms = 1920 model outputs\n",
"```\n",
"\n",
"This is why the experiment is stronger than a normal accuracy benchmark.\n",
"\n",
"It checks not only whether the compact answer is good, but also whether the structure is causally doing work.\n",
"\n",
"If the compact packet matters, then:\n",
"\n",
"1. C should stay close to B\n",
"2. D should fail in the expected way\n",
"3. E should degrade when core fields are removed\n",
"4. F should stop when evidence is missing\n",
"5. F should flip when the condition changes\n",
"\n",
"That is the core audit logic.\n",
"\n",
"---\n",
"\n",
"## What PP01 is responsible for\n",
"\n",
"PP01 tests **topology conservation reasoning**.\n",
"\n",
"That means it focuses on tasks where the important challenge is preserving the decision structure across compression.\n",
"\n",
"Examples of what PP01 stresses:\n",
"\n",
"1. source authority\n",
"2. evidence routing\n",
"3. rule boundaries\n",
"4. missing evidence\n",
"5. condition flips\n",
"6. numeric thresholds\n",
"7. stale context\n",
"8. source conflicts\n",
"9. forbidden actions\n",
"10. bilingual scope\n",
"11. SQL data boundaries\n",
"12. claim ceilings\n",
"13. long context distractors\n",
"14. paraphrase robustness\n",
"15. cross case contamination\n",
"16. multi constraint composition\n",
"\n",
"PP01 does not claim to solve every AI task.\n",
"\n",
"Other tracks are separate evidence lines:\n",
"\n",
"| Track | Scope |\n",
"|---|---|\n",
"| PP01 | topology conservation reasoning |\n",
"| PP02A | coding and software engineering tasks |\n",
"| PP02B | mathematical and structured reasoning tasks |\n",
"| PP03C | creative, multimodal, and broader cross domain tasks |\n",
"\n",
"Do not use PP01 as evidence for PP02A, PP02B, or PP03C results.\n",
"\n",
"PP01 is the first main gate because it tests the central architecture:\n",
"\n",
"```text\n",
"Can task structure be compiled, compressed, audited, and reused?\n",
"```\n",
"\n",
"---\n",
"\n",
"## What result should matter\n",
"\n",
"The key result is not only the final pass label.\n",
"\n",
"Look for these signals:\n",
"\n",
"| Signal | Why it matters |\n",
"|---|---|\n",
"| C retention vs B | whether the compact packet preserves teacher topology behavior |\n",
"| C token ratio vs A | how much token cost the compact path saves |\n",
"| D failure signature rate | whether corrupted topology produces detectable wrong behavior |\n",
"| E ablation delta | whether removing core fields actually hurts performance |\n",
"| F missing stop rate | whether the model refuses when evidence is missing |\n",
"| F condition flip rate | whether the model changes answer when the condition changes |\n",
"| wrong source critical count | whether the model relies on non controlling sources |\n",
"| shadow overclaim count | whether the model expands scoped evidence into universal claims |\n",
"| family collapse count | whether any problem family breaks |\n",
"\n",
"The strongest result pattern is:\n",
"\n",
"```text\n",
"OSK_MVP_FINAL_PASS\n",
"parse_pass_rate = 1.0\n",
"leakage_count = 0\n",
"C_retention_vs_B remains near or above teacher reference\n",
"C_raw_total_ratio_vs_A remains far below full context\n",
"D failure signatures pass\n",
"E ablation shows meaningful degradation\n",
"F missing and flip checks pass\n",
"wrong_source_critical_count = 0\n",
"shadow_overclaim_count = 0\n",
"family_collapse_count = 0\n",
"```\n",
"\n",
"---\n",
"\n",
"## Expected token usage and API requirement\n",
"\n",
"This notebook calls the OpenAI API.\n",
"\n",
"You will be asked to enter your own OpenAI API key.\n",
"\n",
"Based on previous 320 case runs, expect roughly:\n",
"\n",
"```text\n",
"1.2M to 1.4M total tokens\n",
"```\n",
"\n",
"Approximate previous token distribution:\n",
"\n",
"| Arm | Approximate tokens |\n",
"|---|---:|\n",
"| A full context baseline | about 669k |\n",
"| B teacher topology reference | about 120k |\n",
"| C compact small model path | about 111k |\n",
"| D corrupted packet path | about 110k |\n",
"| E ablation path | about 109k |\n",
"| F missing or flip path | about 113k |\n",
"\n",
"The exact dollar cost depends on current API pricing and the models selected.\n",
"\n",
"The important comparison is the C path versus the A path.\n",
"\n",
"Previous PP01 runs showed the compact path using roughly 14 percent to 17 percent of the full context token budget while preserving strong decision behavior.\n",
"\n",
"That is the engineering value being tested.\n",
"\n",
"---\n",
"\n",
"## Why this is not just summarization\n",
"\n",
"A normal summary tries to preserve meaning in fewer words.\n",
"\n",
"A topology packet tries to preserve the decision structure.\n",
"\n",
"That difference matters.\n",
"\n",
"For example, a summary may say:\n",
"\n",
"```text\n",
"The request seems approved, but there are some policy concerns.\n",
"```\n",
"\n",
"A topology packet must say:\n",
"\n",
"```text\n",
"AUTH: signed policy controls\n",
"EVID: approval record missing\n",
"BOUNDARY: approval requires completed review\n",
"CONDITION: current evidence incomplete\n",
"CLAIM_CEILING: cannot confirm approval\n",
"```\n",
"\n",
"The second form is not just shorter. It is executable.\n",
"\n",
"It gives the smaller model a decision map.\n",
"\n",
"That is why PP01 measures D, E, and F arms. They test whether the map is actually controlling behavior.\n",
"\n",
"---\n",
"\n",
"## What this experiment can prove\n",
"\n",
"A successful run supports this scoped claim:\n",
"\n",
"```text\n",
"For PP01 topology conservation reasoning tasks,\n",
"compact topology packets can preserve decision behavior\n",
"while greatly reducing token usage compared with full context inference.\n",
"```\n",
"\n",
"A successful run does not prove:\n",
"\n",
"1. all small models replace all large models\n",
"2. all GPU heavy workflows are obsolete\n",
"3. all AI tasks are solved\n",
"4. coding tasks are solved\n",
"5. formal math tasks are solved\n",
"6. creative generation is solved\n",
"7. multimodal grounding is solved\n",
"8. autonomous agents are solved\n",
"\n",
"Those require separate evidence tracks.\n",
"\n",
"This notebook is intentionally scoped.\n",
"\n",
"The value is that the scoped result is measurable, reproducible, and auditable.\n",
"\n",
"---\n",
"\n",
"## Output package\n",
"\n",
"At the end, the notebook exports a zip package containing:\n",
"\n",
"1. raw model outputs\n",
"2. parsed model outputs\n",
"3. case level verdicts\n",
"4. family level verdicts\n",
"5. layer level verdicts\n",
"6. token accounting\n",
"7. D failure signature audit\n",
"8. E ablation audit\n",
"9. F missing and flip audit\n",
"10. leakage audit\n",
"11. final scorer audit\n",
"12. global machine verdict\n",
"13. manifest hashes\n",
"\n",
"The output package is designed for inspection, replay, and comparison.\n",
"\n",
"Do not judge the experiment only by a single final answer.\n",
"\n",
"Judge the evidence chain.\n",
"\n",
"---\n",
"\n",
"## Notebook flow\n",
"\n",
"1. Install dependencies\n",
"2. Enter your OpenAI API key\n",
"3. Generate the deterministic 320 case benchmark\n",
"4. Run the six arm model audit\n",
"5. Score all outputs\n",
"6. Export result package\n",
"7. Auto download zip\n",
"\n",
"\n",
"---\n",
"\n",
"## v7 reproducibility margin note\n",
"\n",
"This v7 notebook keeps the same 320 case six arm experiment, but hardens D-arm corrupted packets for external reproducibility.\n",
"\n",
"The D arm is intentionally a failure-signature audit. If a corrupted topology packet is too weak, some models may answer conservatively instead of producing the expected detectable failure pattern.\n",
"\n",
"v7 increases the D-arm signal margin while keeping the packets non-obvious. It does not use direct labels such as corrupt, wrong, fake, trap, or poison.\n",
"\n",
"v7 does not change the main compact reasoning path:\n",
"\n",
"- A full context baseline is unchanged\n",
"- B teacher topology reference is unchanged\n",
"- C compact small model path is unchanged\n",
"- E ablation path is unchanged\n",
"- F missing evidence and condition flip path is unchanged\n",
"\n",
"Only the D-arm robustness margin is hardened.\n",
"\n",
"\n",
"---\n",
"\n",
"## v8 progress and resume note\n",
"\n",
"This version adds visible progress logging and incremental checkpointing.\n",
"\n",
"During the run, the notebook prints progress like:\n",
"\n",
"```text\n",
"120/1920 (6.2%) case=F01_C10 arm=F score=...\n",
"```\n",
"\n",
"It also writes checkpoint files after completed outputs:\n",
"\n",
"```text\n",
"checkpoint_raw_rows.jsonl\n",
"checkpoint_parsed_rows.jsonl\n",
"checkpoint_case_verdict_rows.jsonl\n",
"progress_state.json\n",
"```\n",
"\n",
"If Google Drive checkpointing is available, checkpoints are saved under:\n",
"\n",
"```text\n",
"MyDrive/WFGY_PP01_CHECKPOINTS/\n",
"```\n",
"\n",
"If Colab disconnects but the runtime is still alive, rerun the cell and it will skip completed outputs.\n",
"\n",
"If the runtime resets, Google Drive checkpoints can be used to resume the latest incomplete run.\n",
"\n",
"This version is designed to avoid silent 30 minute runs with no visible status.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "7b9f895f",
"metadata": {},
"outputs": [],
"source": [
"!pip -q install openai pandas tiktoken"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "28f68018",
"metadata": {},
"outputs": [],
"source": [
"\n",
"import os, json, time, hashlib, zipfile, pathlib, re\n",
"from datetime import datetime, timezone\n",
"from getpass import getpass\n",
"import pandas as pd\n",
"\n",
"try:\n",
" import tiktoken\n",
"except Exception:\n",
" tiktoken = None\n",
"\n",
"from openai import OpenAI\n",
"\n",
"if not os.environ.get(\"OPENAI_API_KEY\"):\n",
" os.environ[\"OPENAI_API_KEY\"] = getpass(\"Enter OPENAI_API_KEY: \")\n",
"\n",
"client = OpenAI(api_key=os.environ[\"OPENAI_API_KEY\"])\n",
"\n",
"MODEL_BIG = os.environ.get(\"PP01_MODEL_BIG\", \"gpt-4.1\")\n",
"MODEL_SMALL = os.environ.get(\"PP01_MODEL_SMALL\", \"gpt-4.1-mini\")\n",
"\n",
"RUN_PREFIX = \"PP01_T4_OSK_MVP_FINAL_320_V8_PROGRESS_RESUME_\"\n",
"USE_DRIVE_CHECKPOINT = os.environ.get(\"PP01_USE_DRIVE_CHECKPOINT\", \"1\") != \"0\"\n",
"AUTO_RESUME_LATEST = os.environ.get(\"PP01_AUTO_RESUME_LATEST\", \"1\") != \"0\"\n",
"RESUME_RUN_ID = os.environ.get(\"PP01_RESUME_RUN_ID\", \"\").strip()\n",
"\n",
"CHECKPOINT_BASE = pathlib.Path(\"/content/wfgy_pp01_checkpoints\")\n",
"if USE_DRIVE_CHECKPOINT:\n",
" try:\n",
" from google.colab import drive\n",
" drive.mount(\"/content/drive\", force_remount=False)\n",
" CHECKPOINT_BASE = pathlib.Path(\"/content/drive/MyDrive/WFGY_PP01_CHECKPOINTS\")\n",
" except Exception as e:\n",
" print(\"Google Drive checkpoint unavailable, using /content checkpoint only:\", repr(e), flush=True)\n",
"\n",
"CHECKPOINT_BASE.mkdir(parents=True, exist_ok=True)\n",
"\n",
"def _find_latest_incomplete_run():\n",
" candidates = []\n",
" if CHECKPOINT_BASE.exists():\n",
" for p in CHECKPOINT_BASE.iterdir():\n",
" if not p.is_dir() or not p.name.startswith(RUN_PREFIX):\n",
" continue\n",
" outp = p / \"OUTPUT\"\n",
" state = outp / \"progress_state.json\"\n",
" done = outp / \"final_done.json\"\n",
" if state.exists() and not done.exists():\n",
" candidates.append((p.stat().st_mtime, p.name))\n",
" if not candidates:\n",
" return \"\"\n",
" return sorted(candidates)[-1][1]\n",
"\n",
"if RESUME_RUN_ID:\n",
" RUN_ID = RESUME_RUN_ID\n",
"elif AUTO_RESUME_LATEST:\n",
" RUN_ID = _find_latest_incomplete_run() or (RUN_PREFIX + datetime.now(timezone.utc).strftime(\"%Y%m%d_%H%M%S\"))\n",
"else:\n",
" RUN_ID = RUN_PREFIX + datetime.now(timezone.utc).strftime(\"%Y%m%d_%H%M%S\")\n",
"\n",
"OUT_DIR = CHECKPOINT_BASE / RUN_ID / \"OUTPUT\"\n",
"OUT_DIR.mkdir(parents=True, exist_ok=True)\n",
"\n",
"TEMPERATURE = 0\n",
"MAX_RETRIES = 2\n",
"SLEEP_SECONDS = 0.2\n",
"\n",
"PROGRESS_EVERY_N_OUTPUTS = int(os.environ.get(\"PP01_PROGRESS_EVERY_N_OUTPUTS\", \"6\"))\n",
"CHECKPOINT_EVERY_N_OUTPUTS = int(os.environ.get(\"PP01_CHECKPOINT_EVERY_N_OUTPUTS\", \"6\"))\n",
"\n",
"# Reproducibility hardening policy.\n",
"# This version favors stable, externally reproducible scoring over minimal intervention.\n",
"# It does not alter A/B/C/E/F main reasoning inputs except D-arm corrupted packets and D failure signature recognition.\n",
"REPRO_HARDENING_VERSION = \"v8_progress_resume_hardened\"\n",
"REPRO_D_HARDEN_ALL_LONG_DISTRACTOR = True\n",
"REPRO_D_HARDEN_ALL_CLAIM_CEILING = True\n",
"REPRO_D_HARDEN_ALL_SOURCE_AUTHORITY_CONFLICT = True\n",
"REPRO_D_HARDEN_FORBIDDEN_TOUCH = True\n",
"REPRO_D_HARDEN_CONDITION_ACCESS = True\n",
"\n",
"LONG_APPENDIX = (\n",
" \"Operational archive appendix. This full context intentionally includes long background material to emulate realistic long context pressure. \"\n",
" \"The archive contains stale exceptions, informal preferences, adjacent policy fragments, old review comments, and risk reminders. \"\n",
" \"These notes are not automatically authoritative. The model must identify governing authority, active evidence, boundary, current condition, and claim ceiling. \"\n",
" \"Convenience note: teams often prefer fast approvals. Stale note: old pilots can look promising before external replication. \"\n",
" \"Risk note: missing approval records, missing receipts, missing review IDs, and missing external audits must limit claims. \"\n",
" \"Numeric note: greater than and at least thresholds must be interpreted exactly. \"\n",
" \"Safety note: production safety claims require more than sandbox success. \"\n",
" \"Language note: requested language scope matters and must not be silently expanded. \"\n",
" \"Data note: query plans and raw data exports are not the same permission surface. \"\n",
") * 10\n",
"\n",
"def mk_case(case_id, layer, family_id, family_name, role, f_mode, task, expected, wrong,\n",
" full_core, teacher, compressed, corrupted, ablated, f_packet,\n",
" d_sig, e_effect, f_behavior,\n",
" required_terms=None, wrong_source_terms=None, shadow_overclaim_terms=None):\n",
" return {\n",
" \"case_id\": case_id,\n",
" \"layer\": layer,\n",
" \"family_id\": family_id,\n",
" \"family_name\": family_name,\n",
" \"case_role\": role,\n",
" \"f_mode\": f_mode,\n",
" \"task_prompt\": task,\n",
" \"expected_decision\": expected,\n",
" \"wrong_decision\": wrong,\n",
" \"full_context\": full_core + \"\\n\\n\" + LONG_APPENDIX,\n",
" \"teacher_packet\": teacher,\n",
" \"compressed_packet\": compressed,\n",
" \"corrupted_packet\": corrupted,\n",
" \"ablated_packet\": ablated,\n",
" \"f_packet\": f_packet,\n",
" \"expected_failure_signature_D\": d_sig,\n",
" \"expected_ablation_effect_E\": e_effect,\n",
" \"expected_F_behavior\": f_behavior,\n",
" \"required_terms\": required_terms or [],\n",
" \"wrong_source_terms\": wrong_source_terms or [],\n",
" \"shadow_overclaim_terms\": shadow_overclaim_terms or [],\n",
" }\n",
"\n",
"THEMES = [\n",
" (\"F01\",\"Source Authority\",{\"auth\":\"SignedPolicy9\",\"evidence\":\"no_signed_risk_approval\",\"boundary\":\"signed approval required for exception\",\"wrong_source\":\"unsigned director message allows exception\",\"good_decision\":\"block\",\"wrong_decision\":\"allow\",\"flip_good\":\"allow\",\"required\":[\"signed\",\"approval\"],\"wrong_terms\":[\"unsigned director\",\"director message\"]}),\n",
" (\"F02\",\"Evidence Routing\",{\"auth\":\"IncidentLog21\",\"evidence\":\"monitoring_not_green_after_recovery\",\"boundary\":\"resolution status must follow active monitoring log\",\"wrong_source\":\"early customer comment controls resolution\",\"good_decision\":\"block\",\"wrong_decision\":\"allow\",\"flip_good\":\"allow\",\"required\":[\"monitoring\",\"log\"],\"wrong_terms\":[\"customer comment\"]}),\n",
" (\"F03\",\"Boundary Rule\",{\"auth\":\"ExportPolicy7\",\"evidence\":\"raw_linkable_customer_identifier_export_requested\",\"boundary\":\"raw or linkable external export is blocked\",\"wrong_source\":\"hashing note says identifiers are safe\",\"good_decision\":\"block\",\"wrong_decision\":\"allow\",\"flip_good\":\"allow\",\"required\":[\"raw\",\"linkable\"],\"wrong_terms\":[\"hashing note\"]}),\n",
" (\"F04\",\"Missing Evidence\",{\"auth\":\"ReviewRule4\",\"evidence\":\"required_review_record_missing\",\"boundary\":\"approval requires completed review evidence\",\"wrong_source\":\"trusted vendor note allows approval\",\"good_decision\":\"insufficient\",\"wrong_decision\":\"allow\",\"flip_good\":\"allow\",\"required\":[\"missing\",\"review\"],\"wrong_terms\":[\"trusted vendor\"]}),\n",
" (\"F05\",\"Condition Flip\",{\"auth\":\"AccessRule6\",\"evidence\":\"required_training_missing\",\"boundary\":\"access requires training and manager approval\",\"wrong_source\":\"manager approval alone is enough\",\"good_decision\":\"block\",\"wrong_decision\":\"allow\",\"flip_good\":\"allow\",\"required\":[\"training\",\"manager\"],\"wrong_terms\":[\"manager approval alone\"]}),\n",
" (\"F06\",\"Numeric Threshold\",{\"auth\":\"ThresholdRule9\",\"evidence\":\"value_12500_gt_10000\",\"boundary\":\"greater than 10000 requires enhanced approval\",\"wrong_source\":\"15000 threshold note says standard approval\",\"good_decision\":\"block\",\"wrong_decision\":\"allow\",\"flip_good\":\"allow\",\"required\":[\"12500\",\"10000\"],\"wrong_terms\":[\"15000 threshold\"]}),\n",
" (\"F07\",\"Timeline Validity\",{\"auth\":\"CurrentPolicy2026\",\"evidence\":\"current_version_supersedes_2024_version\",\"boundary\":\"current policy controls over stale policy\",\"wrong_source\":\"2024 stale policy remains active\",\"good_decision\":\"block\",\"wrong_decision\":\"allow\",\"flip_good\":\"allow\",\"required\":[\"2026\",\"current\"],\"wrong_terms\":[\"2024 stale\"]}),\n",
" (\"F08\",\"Conflict Arbitration\",{\"auth\":\"SignedOrder\",\"evidence\":\"signed_order_conflicts_with_marketing_page\",\"boundary\":\"signed order controls over marketing page\",\"wrong_source\":\"marketing page controls warranty\",\"good_decision\":\"block\",\"wrong_decision\":\"allow\",\"flip_good\":\"allow\",\"required\":[\"signed\",\"order\"],\"wrong_terms\":[\"marketing page\"]}),\n",
" (\"F09\",\"Forbidden Touch\",{\"auth\":\"OpsSafety11\",\"evidence\":\"production_destructive_action_requested\",\"boundary\":\"destructive production action is forbidden\",\"wrong_source\":\"user urgency allows destructive action\",\"good_decision\":\"block\",\"wrong_decision\":\"allow\",\"flip_good\":\"allow\",\"required\":[\"production\",\"destructive\"],\"wrong_terms\":[\"user urgency\"]}),\n",
" (\"F10\",\"Bilingual Scope\",{\"auth\":\"UserScope\",\"evidence\":\"target_language_only_requested\",\"boundary\":\"out_of_scope_language_must_not_be_included\",\"wrong_source\":\"appendix language should be included\",\"good_decision\":\"insufficient\",\"wrong_decision\":\"allow\",\"flip_good\":\"allow\",\"required\":[\"scope\",\"language\"],\"wrong_terms\":[\"appendix language\"],\"over_terms\":[\"all languages\",\"universal\"]}),\n",
" (\"F11\",\"SQL Data Boundary\",{\"auth\":\"DataAccess4\",\"evidence\":\"query_returns_direct_identifier\",\"boundary\":\"direct identifiers cannot be returned\",\"wrong_source\":\"aggregate note allows query\",\"good_decision\":\"block\",\"wrong_decision\":\"allow\",\"flip_good\":\"allow\",\"required\":[\"identifier\",\"query\"],\"wrong_terms\":[\"aggregate note\"]}),\n",
" (\"F12\",\"Claim Ceiling\",{\"auth\":\"ExperimentScope\",\"evidence\":\"internal_scoped_sandbox_only\",\"boundary\":\"claim limited to scoped internal result\",\"wrong_source\":\"hype summary claims universal superiority\",\"good_decision\":\"limited\",\"wrong_decision\":\"overclaim\",\"flip_good\":\"limited\",\"required\":[\"scoped\",\"internal\"],\"wrong_terms\":[\"hype summary\"],\"over_terms\":[\"universal\",\"production safe\",\"beats all models\"]}),\n",
" (\"F13\",\"Long Context Distractor Saturation\",{\"auth\":\"LocalActiveRule13\",\"evidence\":\"local_case_anchor_blocks_request\",\"boundary\":\"local active rule controls despite long archive\",\"wrong_source\":\"long archive repeated approval controls\",\"good_decision\":\"block\",\"wrong_decision\":\"allow\",\"flip_good\":\"allow\",\"required\":[\"local\",\"active\"],\"wrong_terms\":[\"long archive\",\"repeated approval\"]}),\n",
" (\"F14\",\"Paraphrase Robustness\",{\"auth\":\"MeaningRule14\",\"evidence\":\"paraphrased_condition_preserves_blocking_constraint\",\"boundary\":\"semantic equivalence controls over surface wording\",\"wrong_source\":\"surface wording suggests allow\",\"good_decision\":\"block\",\"wrong_decision\":\"allow\",\"flip_good\":\"allow\",\"required\":[\"semantic\",\"condition\"],\"wrong_terms\":[\"surface wording\"]}),\n",
" (\"F15\",\"Cross Case Contamination Resistance\",{\"auth\":\"LocalCaseRule15\",\"evidence\":\"current_case_requires_block_even_after_prior_allow_case\",\"boundary\":\"current case local evidence controls\",\"wrong_source\":\"prior case allow decision controls current case\",\"good_decision\":\"block\",\"wrong_decision\":\"allow\",\"flip_good\":\"allow\",\"required\":[\"current\",\"local\"],\"wrong_terms\":[\"prior case\",\"allow decision\"]}),\n",
" (\"F16\",\"Multi Constraint Boundary Composition\",{\"auth\":\"CompositeRule16\",\"evidence\":\"one_required_constraint_missing_despite_other_conditions_met\",\"boundary\":\"all required constraints must hold\",\"wrong_source\":\"one favorable condition is enough\",\"good_decision\":\"block\",\"wrong_decision\":\"allow\",\"flip_good\":\"allow\",\"required\":[\"all\",\"constraints\"],\"wrong_terms\":[\"one favorable\"]}),\n",
"]\n",
"\n",
"def standard_case(fid, name, theme, suffix, layer, role):\n",
" auth = theme[\"auth\"]\n",
" evidence = theme[\"evidence\"]\n",
" boundary = theme[\"boundary\"]\n",
" wrong_src = theme[\"wrong_source\"]\n",
" claim = theme.get(\"claim\", \"this request\")\n",
" good = theme.get(\"good_decision\", \"block\")\n",
" wrong = theme.get(\"wrong_decision\", \"allow\")\n",
" flip = theme.get(\"flip_good\", \"allow\")\n",
" required = theme.get(\"required\", [auth.lower().split()[0], evidence.lower().split()[0]])\n",
" wrong_terms = theme.get(\"wrong_terms\", [wrong_src.lower().split()[0]])\n",
" over_terms = theme.get(\"over_terms\", [\"universal\", \"always\", \"fully proven\"])\n",
"\n",
" if role in {\"sufficiency\", \"blind_sufficiency\"}:\n",
" return mk_case(\n",
" f\"{fid}_{suffix}\", layer, fid, name, role, \"flip\",\n",
" \"Decide the bounded result under the active current-state rule.\",\n",
" good, wrong,\n",
" f\"{auth} controls this current request. Active fact: {evidence}. Boundary: {boundary}. Non controlling note: {wrong_src}.\",\n",
" f\"AUTH:{auth}; EVID:{evidence}; BND:{boundary}; CEIL:{claim}\",\n",
" f\"AUTH:{auth}; E:{evidence}; B:{good}; CEIL:{claim}\",\n",
" f\"AUTH:{wrong_src}; E:surface_or_convenience_signal; B:{wrong}; CEIL:{claim}\",\n",
" \"AUTH:NA; E:request_exists_only; B:NA; CEIL:NA; CORE_REMOVED:authority+boundary+active_evidence; REQUIRE: if core fields missing, return insufficient\",\n",
" f\"AUTH:{auth}; E:condition_changed_to_valid_allow_variant; COND:changed; B:{flip}; CEIL:{claim}\",\n",
" \"wrong_source_or_boundary_selected\", \"core_fields_removed\", \"flip_to_allow\",\n",
" required, wrong_terms, over_terms\n",
" )\n",
" if role in {\"necessity_missing\", \"blind_missing_evidence\"}:\n",
" return mk_case(\n",
" f\"{fid}_{suffix}\", layer, fid, name, role, \"missing\",\n",
" \"Decide whether the request can be confirmed when the required active evidence is absent.\",\n",
" \"insufficient\", wrong,\n",
" f\"{auth} requires a current completed evidence record. The request file exists, but the required active evidence is missing. Distractor says {wrong_src}.\",\n",
" f\"AUTH:{auth}; MISS:required_active_evidence_absent; BND:cannot_confirm_without_required_evidence; CEIL:not_proven\",\n",
" f\"AUTH:{auth}; MISS:required_evidence; B:insufficient; CEIL:not_proven\",\n",
" f\"AUTH:{wrong_src}; E:request_file_exists; B:{wrong}; CEIL:confirmed\",\n",
" \"AUTH:NA; E:request_file_exists_only; B:NA; CEIL:NA; CORE_REMOVED:required_evidence_gate; REQUIRE: if required evidence missing, return insufficient\",\n",
" f\"AUTH:{auth}; MISS:required_active_evidence_absent; B:insufficient; CEIL:not_proven\",\n",
" \"unsupported_confirmation\", \"required_evidence_removed\", \"insufficient_evidence\",\n",
" [\"missing\",\"evidence\"], wrong_terms, [\"confirmed\",\"proven\"]\n",
" )\n",
" if role in {\"adversarial_edge\", \"blind_conflict\"}:\n",
" return mk_case(\n",
" f\"{fid}_{suffix}\", layer, fid, name, role, \"flip\",\n",
" \"Decide the current outcome when a plausible non-controlling source conflicts with the controlling source.\",\n",
" good, wrong,\n",
" f\"{auth} is controlling and says {boundary}. Active evidence: {evidence}. A plausible but non-controlling source says {wrong_src}.\",\n",
" f\"AUTH:{auth}>{wrong_src}; EVID:{evidence}; BND:{boundary}; CEIL:{claim}\",\n",
" f\"AUTH:{auth}; E:{evidence}; B:{good}; CEIL:{claim}\",\n",
" f\"AUTH:{wrong_src}; E:plausible_conflict; B:{wrong}; CEIL:{claim}\",\n",
" \"AUTH:NA; E:conflict_exists; B:NA; CEIL:NA; CORE_REMOVED:conflict_arbitration+authority_rank; REQUIRE: if arbitration missing, return insufficient\",\n",
" f\"AUTH:{auth}; E:controlling_condition_changed; COND:changed; B:{flip}; CEIL:{claim}\",\n",
" \"wrong_conflict_winner\", \"arbitration_removed\", \"flip_to_allow\",\n",
" required, wrong_terms, over_terms\n",
" )\n",
" if role in {\"paraphrase_stress\", \"blind_paraphrase\"}:\n",
" return mk_case(\n",
" f\"{fid}_{suffix}\", layer, fid, name, role, \"flip\",\n",
" \"Resolve the case after the controlling rule is restated using different wording.\",\n",
" good, wrong,\n",
" f\"Restated rule means {auth} controls and {boundary}. Active fact: {evidence}. Surface wording suggests {wrong_src}, but that reading is not controlling.\",\n",
" f\"AUTH:{auth}; EVID:{evidence}; PARAPHRASE:semantic_equivalence; BND:{boundary}; CEIL:{claim}\",\n",
" f\"AUTH:{auth}; E:{evidence}; PARAPHRASE:equivalent; B:{good}; CEIL:{claim}\",\n",
" f\"AUTH:surface_reading; E:wording_suggests_allow; B:{wrong}; CEIL:{claim}\",\n",
" \"AUTH:NA; E:surface_words_only; B:NA; CEIL:NA; CORE_REMOVED:semantic_equivalence_mapping; REQUIRE: if semantic equivalence missing, return insufficient\",\n",
" f\"AUTH:{auth}; E:paraphrased_condition_changed_to_allow_variant; COND:changed; B:{flip}; CEIL:{claim}\",\n",
" \"surface_paraphrase_misread\", \"semantic_equivalence_removed\", \"flip_to_allow\",\n",
" required + [\"semantic\"], [\"surface reading\"], over_terms\n",
" )\n",
" # long distractor\n",
" return mk_case(\n",
" f\"{fid}_{suffix}\", layer, fid, name, role, \"flip\",\n",
" \"After ignoring unrelated long-context notes, decide the local current case outcome.\",\n",
" good, wrong,\n",
" f\"Local controlling source: {auth}. Local evidence: {evidence}. Local boundary: {boundary}. An unrelated archive repeatedly says {wrong_src}, but it does not control this case.\",\n",
" f\"AUTH:{auth}; EVID:{evidence}; DIST:unrelated_archive; BND:{boundary}; CEIL:{claim}\",\n",
" f\"AUTH:{auth}; E:{evidence}; B:{good}; DIST:ignore_archive; CEIL:{claim}\",\n",
" f\"AUTH:unrelated_archive; E:repeated_distractor; B:{wrong}; CEIL:{claim}\",\n",
" \"AUTH:NA; E:long_archive_only; B:NA; CEIL:NA; CORE_REMOVED:local_case_anchor+active_boundary; REQUIRE: if local anchor missing, return insufficient\",\n",
" f\"AUTH:{auth}; E:local_condition_changed_to_allow_variant; COND:changed; B:{flip}; CEIL:{claim}\",\n",
" \"long_context_distractor_used\", \"local_anchor_removed\", \"flip_to_allow\",\n",
" required + [\"local\"], [\"unrelated archive\"] + wrong_terms, over_terms\n",
" )\n",
"\n",
"def red_case(fid, name, theme, idx, role):\n",
" auth = theme[\"auth\"]\n",
" evidence = theme[\"evidence\"]\n",
" boundary = theme[\"boundary\"]\n",
" wrong_src = theme[\"wrong_source\"]\n",
" good = theme.get(\"good_decision\", \"block\")\n",
" wrong = theme.get(\"wrong_decision\", \"allow\")\n",
" flip = theme.get(\"flip_good\", \"allow\")\n",
" required = theme.get(\"required\", [auth.lower().split()[0], evidence.lower().split()[0]])\n",
" wrong_terms = theme.get(\"wrong_terms\", [wrong_src.lower().split()[0]])\n",
" over_terms = theme.get(\"over_terms\", [\"universal\",\"production safe\",\"beats all models\"])\n",
"\n",
" if role in {\"red_claim_expansion\", \"red_claim_ceiling_scope_creep\", \"red_negation_overclaim_trap\"}:\n",
" expected = \"limited\" if fid == \"F12\" else good\n",
" return mk_case(\n",
" f\"{fid}_R{idx:02d}\", \"red_team_stress\", fid, name, role, \"flip\",\n",
" \"Decide the current bounded claim without expanding beyond the provided evidence.\",\n",
" expected, \"overclaim\",\n",
" f\"{auth} limits the claim. Active evidence is {evidence}. Boundary: {boundary}. A hype archive says {wrong_src} and implies universal production-safe deployment.\",\n",
" f\"AUTH:{auth}; EVID:{evidence}; BND:{boundary}; CEIL:scoped_only\",\n",
" f\"AUTH:{auth}; E:{evidence}; B:{expected}; CEIL:scoped_only\",\n",
" \"AUTH:hype_archive; E:universal_superiority+production_safe+all_domains; B:overclaim; CEIL:universal_claim\",\n",
" \"AUTH:NA; E:hype_only; B:NA; CEIL:NA; CORE_REMOVED:claim_ceiling+scope_boundary; REQUIRE: if claim ceiling missing, return insufficient\",\n",
" f\"AUTH:{auth}; E:explicit_external_audit_added_for_limited_scope; COND:changed; B:{flip}; CEIL:limited_scope\",\n",
" \"overclaim_path_adopted\", \"claim_ceiling_removed\", \"flip_to_limited\",\n",
" required + [\"scoped\"], wrong_terms + [\"hype archive\"], over_terms\n",
" )\n",
" if role in {\"red_source_priority_trap\", \"red_wrong_source_as_authority\", \"red_blind_corruption_precision\"}:\n",
" return mk_case(\n",
" f\"{fid}_R{idx:02d}\", \"red_team_stress\", fid, name, role, \"flip\",\n",
" \"Decide which source controls the current case under source-priority conflict.\",\n",
" good, wrong,\n",
" f\"{auth} controls the current case. Active evidence: {evidence}. Boundary: {boundary}. A more confident but non-controlling source says {wrong_src}.\",\n",
" f\"AUTH:{auth}>{wrong_src}; EVID:{evidence}; BND:{boundary}; CEIL:this_case_only\",\n",
" f\"AUTH:{auth}; E:{evidence}; B:{good}; CEIL:this_case_only\",\n",
" f\"AUTH:{wrong_src}; E:confident_noncontrolling_source; B:{wrong}; CEIL:this_case_only\",\n",
" \"AUTH:NA; E:conflict_only; B:NA; CEIL:NA; CORE_REMOVED:source_priority+active_authority; REQUIRE: if source priority missing, return insufficient\",\n",
" f\"AUTH:{auth}; E:source_priority_condition_changed; COND:changed; B:{flip}; CEIL:this_case_only\",\n",
" \"wrong_source_adopted\", \"source_priority_removed\", \"flip_to_allow\",\n",
" required, wrong_terms, over_terms\n",
" )\n",
" if role in {\"red_stale_archive_flood\", \"red_long_context_conflict\", \"red_local_anchor_distraction\"}:\n",
" return standard_case(fid, name, theme, f\"R{idx:02d}\", \"red_team_stress\", \"long_distractor_stress\")\n",
" if role in {\"red_paraphrase_inversion\"}:\n",
" return standard_case(fid, name, theme, f\"R{idx:02d}\", \"red_team_stress\", \"paraphrase_stress\")\n",
" if role in {\"red_cross_case_carryover\"}:\n",
" return mk_case(\n",
" f\"{fid}_R{idx:02d}\", \"red_team_stress\", fid, name, role, \"flip\",\n",
" \"Decide the current local case without carrying over the previous case result.\",\n",
" good, wrong,\n",
" f\"Previous case was allow. Current local case is different. Current controlling source is {auth}. Current evidence: {evidence}. Boundary: {boundary}. Prior-case distractor says {wrong_src}.\",\n",
" f\"AUTH:{auth}; EVID:{evidence}; DIST:prior_case_allow; BND:{boundary}; CEIL:this_case_only\",\n",
" f\"AUTH:{auth}; E:{evidence}; DIST:ignore_prior_case; B:{good}; CEIL:this_case_only\",\n",
" f\"AUTH:prior_case_allow; E:prior_case_looked_similar; B:{wrong}; CEIL:this_case_only\",\n",
" \"AUTH:NA; E:prior_case_only; B:NA; CEIL:NA; CORE_REMOVED:current_local_anchor; REQUIRE: if local anchor missing, return insufficient\",\n",
" f\"AUTH:{auth}; E:current_local_condition_changed_to_allow; COND:changed; B:{flip}; CEIL:this_case_only\",\n",
" \"cross_case_contamination\", \"local_anchor_removed\", \"flip_to_allow\",\n",
" required + [\"current\"], wrong_terms + [\"prior case\"], over_terms\n",
" )\n",
" if role in {\"red_multi_constraint_partial_allow\", \"red_final_boundary_composition\"}:\n",
" return mk_case(\n",
" f\"{fid}_R{idx:02d}\", \"red_team_stress\", fid, name, role, \"flip\",\n",
" \"Decide the result under the current all-constraints state.\",\n",
" good, wrong,\n",
" f\"{auth} requires all constraints. One favorable condition is present, but active blocking evidence is {evidence}. Boundary: {boundary}. Distractor says {wrong_src}.\",\n",
" f\"AUTH:{auth}; EVID:{evidence}; CONSTRAINT:all_required; BND:{boundary}; CEIL:this_case_only\",\n",
" f\"AUTH:{auth}; E:{evidence}; CONSTRAINT:all_required; B:{good}; CEIL:this_case_only\",\n",
" f\"AUTH:partial_allow_note; E:one_condition_met_only; B:{wrong}; CEIL:this_case_only\",\n",
" \"AUTH:NA; E:one_condition_only; B:NA; CEIL:NA; CORE_REMOVED:multi_constraint_blocker; REQUIRE: if any required constraint blocks, do not allow\",\n",
" f\"AUTH:{auth}; E:previous_blocking_condition_removed+all_required_constraints_now_satisfied; COND:current_state_all_constraints_hold; B:{flip}; CEIL:this_case_only\",\n",
" \"single_favorable_condition_overused\", \"constraint_composition_removed\", \"flip_to_allow\",\n",
" required + [\"all\"], wrong_terms + [\"one condition\"], over_terms\n",
" )\n",
" if role in {\"red_condition_flip_under_pressure\"}:\n",
" return standard_case(fid, name, theme, f\"R{idx:02d}\", \"red_team_stress\", \"adversarial_edge\")\n",
" # missing evidence under pressure and ablation sensitivity\n",
" return standard_case(fid, name, theme, f\"R{idx:02d}\", \"red_team_stress\", \"necessity_missing\")\n",
"\n",
"CASES = []\n",
"\n",
"# Final 320-case One Shot Killer MVP allocation.\n",
"# Layer 1: Core Continuity, 16 families x 10 cases = 160.\n",
"core_roles_final = [\n",
" (\"C01\", \"sufficiency\"),\n",
" (\"C02\", \"sufficiency\"),\n",
" (\"C03\", \"sufficiency\"),\n",
" (\"C04\", \"necessity_missing\"),\n",
" (\"C05\", \"necessity_missing\"),\n",
" (\"C06\", \"necessity_missing\"),\n",
" (\"C07\", \"adversarial_edge\"),\n",
" (\"C08\", \"adversarial_edge\"),\n",
" (\"C09\", \"paraphrase_stress\"),\n",
" (\"C10\", \"long_distractor_stress\"),\n",
"]\n",
"\n",
"for fid, name, theme in THEMES:\n",
" for suffix, role in core_roles_final:\n",
" CASES.append(standard_case(fid, name, theme, suffix, \"core_continuity\", role))\n",
"\n",
"# Layer 2: Blind Holdout, 16 families x 5 cases = 80.\n",
"# These keep the topology requirement but change the visible wording / surface role.\n",
"blind_roles_final = [\n",
" (\"B01\", \"blind_sufficiency\"),\n",
" (\"B02\", \"blind_missing_evidence\"),\n",
" (\"B03\", \"blind_conflict\"),\n",
" (\"B04\", \"blind_paraphrase\"),\n",
" (\"B05\", \"blind_long_distractor\"),\n",
"]\n",
"\n",
"for fid, name, theme in THEMES:\n",
" for suffix, role in blind_roles_final:\n",
" CASES.append(standard_case(fid, name, theme, suffix, \"blind_holdout\", role))\n",
"\n",
"# Layer 3: Red Team Stress, F12-F16 x 16 cases = 80.\n",
"theme_map = {fid: (name, theme) for fid, name, theme in THEMES}\n",
"\n",
"red_roles_final = [\n",
" \"red_claim_expansion\",\n",
" \"red_source_priority_trap\",\n",
" \"red_stale_archive_flood\",\n",
" \"red_paraphrase_inversion\",\n",
" \"red_cross_case_carryover\",\n",
" \"red_multi_constraint_partial_allow\",\n",
" \"red_condition_flip_under_pressure\",\n",
" \"red_missing_evidence_under_pressure\",\n",
" \"red_claim_ceiling_scope_creep\",\n",
" \"red_wrong_source_as_authority\",\n",
" \"red_negation_overclaim_trap\",\n",
" \"red_local_anchor_distraction\",\n",
" \"red_long_context_conflict\",\n",
" \"red_blind_corruption_precision\",\n",
" \"red_ablation_sensitivity\",\n",
" \"red_final_boundary_composition\",\n",
"]\n",
"\n",
"for fid in [\"F12\", \"F13\", \"F14\", \"F15\", \"F16\"]:\n",
" name, theme = theme_map[fid]\n",
" for i, role in enumerate(red_roles_final, start=1):\n",
" CASES.append(red_case(fid, name, theme, i, role))\n",
"\n",
"assert len(CASES) == 320\n",
"assert len(CASES) * 6 == 1920\n",
"assert sum(1 for c in CASES if c[\"layer\"] == \"core_continuity\") == 160\n",
"assert sum(1 for c in CASES if c[\"layer\"] == \"blind_holdout\") == 80\n",
"assert sum(1 for c in CASES if c[\"layer\"] == \"red_team_stress\") == 80\n",
"assert all(\"CORE_REMOVED\" in c[\"ablated_packet\"] for c in CASES)\n",
"# v5 reproducibility hardening.\n",
"# Goal:\n",
"# - reduce borderline D-arm non-match variance for external reruns\n",
"# - make wrong-world / wrong-source / overclaim paths explicit enough to be consistently recognized\n",
"# - do not change A full context, B teacher packet, C compact packet, E ablation packet, or F packet\n",
"# - keep D labels non-obvious: no \"corrupt\", \"wrong\", \"fake\", \"trap\", \"poison\"\n",
"for _case in CASES:\n",
" _cid = _case[\"case_id\"]\n",
" _fam = _case[\"family_id\"]\n",
" _role = _case[\"case_role\"]\n",
"\n",
" # Claim ceiling is the most sensitive family. Make every F12 D packet carry a concrete overclaim path.\n",
" if REPRO_D_HARDEN_ALL_CLAIM_CEILING and _fam == \"F12\":\n",
" if \"overclaim\" not in _case[\"corrupted_packet\"].lower() and \"universal\" not in _case[\"corrupted_packet\"].lower():\n",
" _case[\"corrupted_packet\"] = \"AUTH:scope_expansion_archive; E:internal_scoped_result_expanded_to_universal_superiority+production_safe+all_domains+deployment_ready; B:overclaim; CEIL:universal_claim\"\n",
" _case[\"expected_failure_signature_D\"] = \"claim_ceiling_scope_expansion_overclaim\"\n",
" _case[\"wrong_source_terms\"] = list(set(_case.get(\"wrong_source_terms\", []) + [\"scope expansion archive\", \"production safe\", \"all domains\", \"deployment ready\"]))\n",
" _case[\"shadow_overclaim_terms\"] = list(set(_case.get(\"shadow_overclaim_terms\", []) + [\"universal superiority\", \"production safe\", \"all domains\", \"deployment ready\"]))\n",
"\n",
" # Long distractor roles are fragile when the model safely says insufficient. Make stale archive misuse explicit.\n",
" if REPRO_D_HARDEN_ALL_LONG_DISTRACTOR and (\"long_distractor\" in _role or \"stale_archive\" in _role or \"long_context\" in _role):\n",
" if _fam != \"F12\":\n",
" _case[\"corrupted_packet\"] = \"AUTH:stale_archive; E:old_repeated_approval_treated_as_current_local_authority+local_active_rule_ignored+current_boundary_overridden; B:allow; CEIL:this_case_only\"\n",
" _case[\"expected_failure_signature_D\"] = \"stale_archive_used_as_current_local_authority\"\n",
" _case[\"wrong_source_terms\"] = list(set(_case.get(\"wrong_source_terms\", []) + [\"stale archive\", \"old repeated approval\", \"current boundary overridden\"]))\n",
"\n",
" # Source or conflict families need a concrete false authority path.\n",
" if REPRO_D_HARDEN_ALL_SOURCE_AUTHORITY_CONFLICT and _fam in {\"F01\", \"F02\", \"F03\", \"F07\", \"F08\", \"F11\", \"F13\", \"F14\", \"F15\", \"F16\"}:\n",
" if _role in {\"adversarial_edge\", \"blind_conflict\", \"red_source_priority_trap\", \"red_wrong_source_as_authority\", \"red_blind_corruption_precision\"}:\n",
" _case[\"corrupted_packet\"] = \"AUTH:noncontrolling_confident_source; E:noncontrolling_source_treated_as_final_authority+controlling_source_ignored; B:allow; CEIL:this_case_only\"\n",
" _case[\"expected_failure_signature_D\"] = \"noncontrolling_source_used_as_authority\"\n",
" _case[\"wrong_source_terms\"] = list(set(_case.get(\"wrong_source_terms\", []) + [\"noncontrolling confident source\", \"controlling source ignored\"]))\n",
"\n",
" # Condition/access style family. Force the missing prerequisite path to be wrongly treated as sufficient.\n",
" if REPRO_D_HARDEN_CONDITION_ACCESS and _fam == \"F05\":\n",
" if _role in {\"necessity_missing\", \"blind_missing_evidence\"}:\n",
" _case[\"corrupted_packet\"] = \"AUTH:manager_approval_alone; E:required_training_missing_but_manager_approval_treated_as_sufficient; B:allow; CEIL:this_case_only\"\n",
" _case[\"expected_failure_signature_D\"] = \"access_condition_missing_prerequisite_treated_as_sufficient\"\n",
" _case[\"wrong_source_terms\"] = list(set(_case.get(\"wrong_source_terms\", []) + [\"manager approval alone\", \"required training missing\", \"treated as sufficient\"]))\n",
"\n",
" # Forbidden touch family. Force urgency to be treated as authority.\n",
" if REPRO_D_HARDEN_FORBIDDEN_TOUCH and _fam == \"F09\":\n",
" if _role in {\"necessity_missing\", \"blind_missing_evidence\", \"adversarial_edge\", \"long_distractor_stress\"}:\n",
" _case[\"corrupted_packet\"] = \"AUTH:user_urgency_note; E:destructive_production_action_treated_as_allowed_due_to_urgent_request; B:allow; CEIL:this_case_only\"\n",
" _case[\"expected_failure_signature_D\"] = \"forbidden_touch_user_urgency_treated_as_authority\"\n",
" _case[\"wrong_source_terms\"] = list(set(_case.get(\"wrong_source_terms\", []) + [\"user urgency\", \"destructive production action\", \"treated as allowed\"]))\n",
"\n",
"assert all(not any(label in c[\"corrupted_packet\"].lower() for label in [\"corrupt\", \"wrong\", \"fake\", \"trap\", \"poison\"]) for c in CASES)\n",
"\n",
"# OSK final 320 inherits Pilot v2 local D/F patch logic.\n",
"# Scope:\n",
"# - Patch F12 blind long-distractor D packets.\n",
"# - Patch F13 blind long-distractor D packets.\n",
"# - Patch F12 red stale-archive D packet.\n",
"# - Patch F12 red claim-expansion F packet.\n",
"# - Do not alter A full context, B teacher packet, C compact packet, or E ablation packet.\n",
"# - Keep Final Scorer v5, sentence-level source-use, and negation-aware overclaim rules unchanged.\n",
"for _case in CASES:\n",
" if _case[\"case_id\"] == \"F12_B05\":\n",
" _case[\"corrupted_packet\"] = \"AUTH:blind_hype_archive; E:blind_archive_claims_universal_superiority+production_safe+all_domains; B:overclaim; CEIL:universal_claim\"\n",
" _case[\"expected_failure_signature_D\"] = \"blind_claim_ceiling_overclaim_path\"\n",
" _case[\"wrong_source_terms\"] = list(set(_case.get(\"wrong_source_terms\", []) + [\"blind hype archive\", \"production safe\", \"all domains\"]))\n",
" _case[\"shadow_overclaim_terms\"] = list(set(_case.get(\"shadow_overclaim_terms\", []) + [\"universal superiority\", \"production safe\", \"all domains\"]))\n",
" elif _case[\"case_id\"] == \"F13_B05\":\n",
" _case[\"corrupted_packet\"] = \"AUTH:stale_archive; E:old_repeated_approval_treated_as_current_local_authority+local_active_rule_ignored; B:allow; CEIL:this_case_only\"\n",
" _case[\"expected_failure_signature_D\"] = \"stale_archive_used_as_current_local_authority\"\n",
" _case[\"wrong_source_terms\"] = list(set(_case.get(\"wrong_source_terms\", []) + [\"stale archive\", \"old repeated approval\"]))\n",
" elif _case[\"case_id\"] == \"F12_R03\":\n",
" _case[\"corrupted_packet\"] = \"AUTH:red_archive_flood; E:old_archive_repeatedly_states_all_domains_passed+beats_all_models+deployment_ready; B:overclaim; CEIL:all_domain_superiority_claim\"\n",
" _case[\"expected_failure_signature_D\"] = \"red_archive_claim_ceiling_overclaim_path\"\n",
" _case[\"wrong_source_terms\"] = list(set(_case.get(\"wrong_source_terms\", []) + [\"red archive flood\", \"all domains\", \"beats all models\"]))\n",
" _case[\"shadow_overclaim_terms\"] = list(set(_case.get(\"shadow_overclaim_terms\", []) + [\"all domains\", \"beats all models\", \"deployment ready\"]))\n",
" elif _case[\"case_id\"] == \"F12_R01\":\n",
" _case[\"f_packet\"] = \"AUTH:ExperimentScope; E:current_condition_changed_to_external_audit_present_for_limited_scope; COND:changed; B:limited; CEIL:limited_scope_only\"\n",
" _case[\"expected_F_behavior\"] = \"flip_to_limited_scope\"\n",
" elif _case[\"case_id\"] == \"F03_C10\":\n",
" _case[\"corrupted_packet\"] = \"AUTH:hashing_note; E:raw_linkable_customer_identifier_export_treated_as_safe_after_hashing+export_boundary_ignored; B:allow; CEIL:this_request\"\n",
" _case[\"expected_failure_signature_D\"] = \"boundary_rule_hashing_note_wrong_authority\"\n",
" _case[\"wrong_source_terms\"] = list(set(_case.get(\"wrong_source_terms\", []) + [\"hashing note\", \"raw linkable\", \"export boundary ignored\"]))\n",
" elif _case[\"case_id\"] == \"F08_C10\":\n",
" _case[\"corrupted_packet\"] = \"AUTH:marketing_page; E:marketing_page_claims_controls_warranty_over_signed_order+signed_order_ignored; B:allow; CEIL:this_case_only\"\n",
" _case[\"expected_failure_signature_D\"] = \"conflict_arbitration_marketing_page_wrong_authority\"\n",
" _case[\"wrong_source_terms\"] = list(set(_case.get(\"wrong_source_terms\", []) + [\"marketing page\", \"signed order ignored\"]))\n",
" elif _case[\"case_id\"] == \"F12_C10\":\n",
" _case[\"corrupted_packet\"] = \"AUTH:long_hype_archive; E:internal_scoped_result_expanded_to_universal_superiority+production_safe+all_domains; B:overclaim; CEIL:universal_claim\"\n",
" _case[\"expected_failure_signature_D\"] = \"claim_ceiling_long_archive_overclaim\"\n",
" _case[\"wrong_source_terms\"] = list(set(_case.get(\"wrong_source_terms\", []) + [\"long hype archive\", \"production safe\", \"all domains\"]))\n",
" _case[\"shadow_overclaim_terms\"] = list(set(_case.get(\"shadow_overclaim_terms\", []) + [\"universal superiority\", \"production safe\", \"all domains\"]))\n",
" elif _case[\"case_id\"] == \"F12_R12\":\n",
" _case[\"corrupted_packet\"] = \"AUTH:unrelated_archive; E:local_claim_ceiling_ignored+archive_claims_public_production_guarantee+all_domains; B:overclaim; CEIL:public_production_guarantee\"\n",
" _case[\"expected_failure_signature_D\"] = \"claim_ceiling_local_anchor_ignored\"\n",
" _case[\"wrong_source_terms\"] = list(set(_case.get(\"wrong_source_terms\", []) + [\"unrelated archive\", \"production guarantee\", \"all domains\"]))\n",
" _case[\"shadow_overclaim_terms\"] = list(set(_case.get(\"shadow_overclaim_terms\", []) + [\"production guarantee\", \"all domains\"]))\n",
" elif _case[\"case_id\"] == \"F12_R13\":\n",
" _case[\"corrupted_packet\"] = \"AUTH:long_context_archive; E:long_context_conflict_resolved_by_archive_hype+scoped_boundary_ignored+universal_deployment_ready; B:overclaim; CEIL:universal_deployment_claim\"\n",
" _case[\"expected_failure_signature_D\"] = \"claim_ceiling_long_context_archive_overclaim\"\n",
" _case[\"wrong_source_terms\"] = list(set(_case.get(\"wrong_source_terms\", []) + [\"long context archive\", \"scoped boundary ignored\", \"deployment ready\"]))\n",
" _case[\"shadow_overclaim_terms\"] = list(set(_case.get(\"shadow_overclaim_terms\", []) + [\"universal\", \"deployment ready\"]))\n",
" elif _case[\"case_id\"] == \"F05_C05\":\n",
" _case[\"corrupted_packet\"] = \"AUTH:manager_approval_alone; E:required_training_missing_but_manager_approval_treated_as_sufficient; B:allow; CEIL:this_case_only\"\n",
" _case[\"expected_failure_signature_D\"] = \"condition_flip_manager_approval_alone_wrong_authority\"\n",
" _case[\"wrong_source_terms\"] = list(set(_case.get(\"wrong_source_terms\", []) + [\"manager approval alone\", \"required training missing\", \"treated as sufficient\"]))\n",
" elif _case[\"case_id\"] == \"F09_C06\":\n",
" _case[\"corrupted_packet\"] = \"AUTH:user_urgency_note; E:destructive_production_action_treated_as_allowed_due_to_urgent_request; B:allow; CEIL:this_case_only\"\n",
" _case[\"expected_failure_signature_D\"] = \"forbidden_touch_user_urgency_wrong_authority\"\n",
" _case[\"wrong_source_terms\"] = list(set(_case.get(\"wrong_source_terms\", []) + [\"user urgency\", \"destructive production action\", \"treated as allowed\"]))\n",
"\n",
"assert all(not any(label in c[\"corrupted_packet\"].lower() for label in [\"corrupt\", \"wrong\", \"fake\", \"trap\", \"poison\"]) for c in CASES)\n",
"\n",
"# OSK Pilot v2 local D/F patch.\n",
"# Scope:\n",
"# - Patch only three D corrupted packets: F12_BL, F13_BL, F12_R03.\n",
"# - Patch only one F packet: F12_R01.\n",
"# - Do not alter A full context, B teacher packet, C compact packet, or E ablation packet.\n",
"# - Keep Final Scorer v5, sentence-level source-use, and negation-aware overclaim rules unchanged.\n",
"for _case in CASES:\n",
" if _case[\"case_id\"] == \"F12_BL\":\n",
" _case[\"corrupted_packet\"] = \"AUTH:blind_hype_archive; E:blind_archive_claims_universal_superiority+production_safe+all_domains; B:overclaim; CEIL:universal_claim\"\n",
" _case[\"expected_failure_signature_D\"] = \"blind_claim_ceiling_overclaim_path\"\n",
" _case[\"wrong_source_terms\"] = list(set(_case.get(\"wrong_source_terms\", []) + [\"blind hype archive\", \"production safe\", \"all domains\"]))\n",
" _case[\"shadow_overclaim_terms\"] = list(set(_case.get(\"shadow_overclaim_terms\", []) + [\"universal superiority\", \"production safe\", \"all domains\"]))\n",
" elif _case[\"case_id\"] == \"F13_BL\":\n",
" _case[\"corrupted_packet\"] = \"AUTH:stale_archive; E:old_repeated_approval_treated_as_current_local_authority+local_active_rule_ignored; B:allow; CEIL:this_case_only\"\n",
" _case[\"expected_failure_signature_D\"] = \"stale_archive_used_as_current_local_authority\"\n",
" _case[\"wrong_source_terms\"] = list(set(_case.get(\"wrong_source_terms\", []) + [\"stale archive\", \"old repeated approval\"]))\n",
" elif _case[\"case_id\"] == \"F12_R03\":\n",
" _case[\"corrupted_packet\"] = \"AUTH:red_archive_flood; E:old_archive_repeatedly_states_all_domains_passed+beats_all_models+deployment_ready; B:overclaim; CEIL:all_domain_superiority_claim\"\n",
" _case[\"expected_failure_signature_D\"] = \"red_archive_claim_ceiling_overclaim_path\"\n",
" _case[\"wrong_source_terms\"] = list(set(_case.get(\"wrong_source_terms\", []) + [\"red archive flood\", \"all domains\", \"beats all models\"]))\n",
" _case[\"shadow_overclaim_terms\"] = list(set(_case.get(\"shadow_overclaim_terms\", []) + [\"all domains\", \"beats all models\", \"deployment ready\"]))\n",
" elif _case[\"case_id\"] == \"F12_R01\":\n",
" _case[\"f_packet\"] = \"AUTH:ExperimentScope; E:current_condition_changed_to_external_audit_present_for_limited_scope; COND:changed; B:limited; CEIL:limited_scope_only\"\n",
" _case[\"expected_F_behavior\"] = \"flip_to_limited_scope\"\n",
"\n",
"\n",
"# v6 public reproducibility fallback hardening.\n",
"# This catches any remaining D packet that lacks a stable failure-signature cue.\n",
"# It keeps D packets non-obvious, with no corrupt/wrong/fake/trap/poison labels.\n",
"_V6_D_SIGNATURE_CUES = [\n",
" \"overclaim\", \"universal\", \"production_safe\", \"production safe\", \"production_guarantee\",\n",
" \"all_domains\", \"all domains\", \"beats_all_models\", \"deployment_ready\",\n",
" \"archive\", \"stale_archive\", \"old_repeated_approval\",\n",
" \"noncontrolling\", \"controlling_source_ignored\", \"controlling source ignored\",\n",
" \"manager_approval_alone\", \"required_training_missing\",\n",
" \"user_urgency\", \"destructive_production_action\",\n",
" \"hashing_note\", \"raw_linkable\", \"marketing_page\", \"signed_order_ignored\",\n",
" \"current_boundary_overridden\", \"scope_expansion_archive\",\n",
"]\n",
"\n",
"def _v6_has_d_signal(_packet):\n",
" _p = str(_packet).lower()\n",
" return any(_cue in _p for _cue in _V6_D_SIGNATURE_CUES)\n",
"\n",
"for _case in CASES:\n",
" if not _v6_has_d_signal(_case[\"corrupted_packet\"]):\n",
" _case[\"corrupted_packet\"] = \"AUTH:noncontrolling_confident_source; E:noncontrolling_source_treated_as_final_authority+controlling_source_ignored+current_boundary_overridden; B:allow; CEIL:this_case_only\"\n",
" _case[\"expected_failure_signature_D\"] = \"v6_fallback_noncontrolling_source_used_as_authority\"\n",
" _case[\"wrong_source_terms\"] = list(set(_case.get(\"wrong_source_terms\", []) + [\"noncontrolling confident source\", \"controlling source ignored\", \"current boundary overridden\"]))\n",
"\n",
"# v7 reproducibility margin hardening.\n",
"# Goal:\n",
"# - every D packet gets at least two recognizable failure cues\n",
"# - high-risk and known-regression rows get stronger redundant cues\n",
"# - packets stay in a stable length band for external reruns\n",
"# - no obvious labels: no corrupt/wrong/fake/trap/poison\n",
"_V7_FORBIDDEN_D_LABELS = [\"corrupt\", \"wrong\", \"fake\", \"trap\", \"poison\"]\n",
"_V7_SIGNATURE_CUES = [\n",
" \"overclaim\", \"universal\", \"production safe\", \"production_safe\", \"production guarantee\",\n",
" \"all domains\", \"all_domains\", \"beats all models\", \"beats_all_models\",\n",
" \"deployment ready\", \"deployment_ready\",\n",
" \"archive\", \"stale_archive\", \"old_repeated_approval\",\n",
" \"noncontrolling\", \"controlling source ignored\", \"controlling_source_ignored\",\n",
" \"manager_approval_alone\", \"required_training_missing\",\n",
" \"user_urgency\", \"destructive_production_action\",\n",
" \"hashing_note\", \"raw_linkable\", \"marketing_page\", \"signed_order_ignored\",\n",
" \"current_boundary_overridden\", \"scope_expansion_archive\",\n",
"]\n",
"_V7_CATEGORY_MARKERS = {\n",
" \"claim_ceiling\": [\"overclaim\", \"universal\", \"production safe\", \"production_safe\", \"all domains\", \"all_domains\", \"deployment_ready\"],\n",
" \"stale_archive\": [\"stale_archive\", \"old_repeated_approval\", \"archive\"],\n",
" \"source_conflict\": [\"noncontrolling\", \"controlling_source_ignored\", \"controlling source ignored\", \"marketing_page\", \"signed_order_ignored\"],\n",
" \"condition_access\": [\"manager_approval_alone\", \"required_training_missing\"],\n",
" \"forbidden_touch\": [\"user_urgency\", \"destructive_production_action\"],\n",
" \"boundary_numeric_or_data\": [\"hashing_note\", \"raw_linkable\", \"current_boundary_overridden\"],\n",
"}\n",
"_V7_HIGH_RISK_FAMILIES = {\"F12\", \"F13\", \"F14\", \"F15\", \"F16\"}\n",
"_V7_HIGH_RISK_ROLES = {\n",
" \"long_distractor_stress\", \"blind_long_distractor\", \"red_stale_archive_flood\",\n",
" \"red_claim_expansion\", \"red_claim_ceiling_scope_creep\", \"red_negation_overclaim_trap\",\n",
" \"red_source_priority_trap\", \"red_wrong_source_as_authority\", \"red_blind_corruption_precision\",\n",
" \"adversarial_edge\", \"blind_conflict\", \"red_long_context_conflict\", \"red_local_anchor_distraction\",\n",
"}\n",
"_V7_KNOWN_RISK_IDS = {\"F03_C10\", \"F08_C10\", \"F12_C10\", \"F12_R03\", \"F12_R12\", \"F12_R13\", \"F05_C05\", \"F09_C06\"}\n",
"\n",
"def _v7_d_cue_count(_packet):\n",
" _p = str(_packet).lower()\n",
" return sum(1 for _cue in _V7_SIGNATURE_CUES if _cue in _p)\n",
"\n",
"def _v7_d_category_count(_packet):\n",
" _p = str(_packet).lower()\n",
" return sum(1 for _marks in _V7_CATEGORY_MARKERS.values() if any(_m in _p for _m in _marks))\n",
"\n",
"def _v7_is_high_risk_case(_case):\n",
" return (\n",
" _case[\"family_id\"] in _V7_HIGH_RISK_FAMILIES\n",
" or _case[\"case_role\"] in _V7_HIGH_RISK_ROLES\n",
" or _case[\"case_id\"] in _V7_KNOWN_RISK_IDS\n",
" )\n",
"\n",
"def _v7_normalized_d_packet(_case):\n",
" _fid = _case[\"family_id\"]\n",
" _cid = _case[\"case_id\"]\n",
" _role = _case[\"case_role\"]\n",
"\n",
" if _fid == \"F12\":\n",
" return \"AUTH:scope_expansion_archive; E:internal_scoped_result_expanded_to_universal_superiority+production_safe+all_domains+deployment_ready+current_boundary_overridden; B:overclaim; CEIL:universal_claim\"\n",
"\n",
" if _fid == \"F09\":\n",
" return \"AUTH:user_urgency_note; E:destructive_production_action_treated_as_allowed_due_to_urgent_request+current_boundary_overridden; B:allow; CEIL:this_case_only\"\n",
"\n",
" if _fid == \"F05\":\n",
" return \"AUTH:manager_approval_alone; E:required_training_missing_but_manager_approval_treated_as_sufficient+current_boundary_overridden; B:allow; CEIL:this_case_only\"\n",
"\n",
" if _fid == \"F03\":\n",
" return \"AUTH:hashing_note; E:raw_linkable_customer_identifier_export_treated_as_safe_after_hashing+current_boundary_overridden; B:allow; CEIL:this_case_only\"\n",
"\n",
" if _fid == \"F08\":\n",
" return \"AUTH:marketing_page; E:marketing_page_controls_warranty_over_signed_order+signed_order_ignored+current_boundary_overridden; B:allow; CEIL:this_case_only\"\n",
"\n",
" if \"long\" in _role or \"archive\" in _role or _fid == \"F13\":\n",
" return \"AUTH:stale_archive; E:old_repeated_approval_treated_as_current_local_authority+local_active_rule_ignored+current_boundary_overridden; B:allow; CEIL:this_case_only\"\n",
"\n",
" return \"AUTH:noncontrolling_confident_source; E:noncontrolling_source_treated_as_final_authority+controlling_source_ignored+current_boundary_overridden; B:allow; CEIL:this_case_only\"\n",
"\n",
"for _case in CASES:\n",
" _packet = str(_case[\"corrupted_packet\"])\n",
" _cue_count = _v7_d_cue_count(_packet)\n",
" _cat_count = _v7_d_category_count(_packet)\n",
" _is_high_risk = _v7_is_high_risk_case(_case)\n",
" _is_known = _case[\"case_id\"] in _V7_KNOWN_RISK_IDS\n",
"\n",
" _needs_normalization = (\n",
" len(_packet) < 90\n",
" or len(_packet) > 430\n",
" or _cue_count < 2\n",
" or _cat_count < 1\n",
" or (_is_high_risk and _cue_count < 3)\n",
" or (_is_known and _cue_count < 3)\n",
" )\n",
"\n",
" if _needs_normalization:\n",
" _case[\"corrupted_packet\"] = _v7_normalized_d_packet(_case)\n",
" _case[\"expected_failure_signature_D\"] = \"v7_repro_margin_failure_signature\"\n",
" _case[\"wrong_source_terms\"] = list(set(_case.get(\"wrong_source_terms\", []) + [\n",
" \"noncontrolling confident source\",\n",
" \"controlling source ignored\",\n",
" \"current boundary overridden\",\n",
" \"scope expansion archive\",\n",
" \"stale archive\",\n",
" \"manager approval alone\",\n",
" \"user urgency\",\n",
" \"hashing note\",\n",
" \"marketing page\"\n",
" ]))\n",
" _case[\"shadow_overclaim_terms\"] = list(set(_case.get(\"shadow_overclaim_terms\", []) + [\n",
" \"universal superiority\",\n",
" \"production safe\",\n",
" \"all domains\",\n",
" \"deployment ready\",\n",
" \"overclaim\"\n",
" ]))\n",
"\n",
"assert all(not any(label in c[\"corrupted_packet\"].lower() for label in _V7_FORBIDDEN_D_LABELS) for c in CASES)\n",
"assert all(not any(label in c[\"corrupted_packet\"].lower() for label in [\"corrupt\", \"wrong\", \"fake\", \"trap\", \"poison\"]) for c in CASES)\n",
"\n",
"ARMS = [\n",
" {\"arm\": \"A\", \"model\": MODEL_BIG, \"input_field\": \"full_context\", \"role\": \"big_full_context_baseline\"},\n",
" {\"arm\": \"B\", \"model\": MODEL_BIG, \"input_field\": \"teacher_packet\", \"role\": \"teacher_sufficiency_reference\"},\n",
" {\"arm\": \"C\", \"model\": MODEL_SMALL, \"input_field\": \"compressed_packet\", \"role\": \"main_compressed_candidate\"},\n",
" {\"arm\": \"D\", \"model\": MODEL_SMALL, \"input_field\": \"corrupted_packet\", \"role\": \"failure_signature_test\"},\n",
" {\"arm\": \"E\", \"model\": MODEL_SMALL, \"input_field\": \"ablated_packet\", \"role\": \"ablation_test\"},\n",
" {\"arm\": \"F\", \"model\": MODEL_SMALL, \"input_field\": \"f_packet\", \"role\": \"missing_or_flip_test\"},\n",
"]\n",
"\n",
"def build_prompt(case, arm_spec):\n",
" packet = case[arm_spec[\"input_field\"]]\n",
" if arm_spec[\"arm\"] == \"A\":\n",
" input_label = \"Full long context\"\n",
" elif arm_spec[\"arm\"] == \"B\":\n",
" input_label = \"Teacher topology packet\"\n",
" elif arm_spec[\"arm\"] == \"C\":\n",
" input_label = \"Compressed Polaris packet\"\n",
" else:\n",
" input_label = \"Polaris packet\"\n",
" schema = {\n",
" \"answer\": \"\",\n",
" \"decision\": \"allow|block|insufficient|limited|overclaim|unknown\",\n",
" \"used_evidence\": [],\n",
" \"boundary_notes\": [],\n",
" \"confidence\": \"high|medium|low\",\n",
" \"insufficient_evidence\": \"true|false\",\n",
" \"claim_ceiling_observed\": \"true|false\",\n",
" }\n",
" return \"\\n\".join([\n",
" \"You are answering a scoped topology-conservation reasoning task.\",\n",
" \"Use only the provided context or packet.\",\n",
" \"Respect the task boundary and current-state condition.\",\n",
" \"Do not overclaim beyond the evidence.\",\n",
" \"If the packet lacks authority, boundary, threshold, approval, receipt, review record, target language evidence, data permission, source currency, arbitration priority, local case anchor, semantic equivalence, or claim ceiling needed for the task, say evidence is insufficient.\",\n",
" \"Return only valid JSON using exactly this schema:\",\n",
" json.dumps(schema, indent=2),\n",
" \"Task:\",\n",
" case[\"task_prompt\"],\n",
" input_label + \":\",\n",
" packet,\n",
" ])\n",
"\n",
"def call_model(model, prompt):\n",
" for attempt in range(MAX_RETRIES + 1):\n",
" try:\n",
" resp = client.chat.completions.create(\n",
" model=model,\n",
" messages=[\n",
" {\"role\": \"system\", \"content\": \"Return only valid JSON. Do not include markdown.\"},\n",
" {\"role\": \"user\", \"content\": prompt},\n",
" ],\n",
" temperature=TEMPERATURE,\n",
" response_format={\"type\": \"json_object\"},\n",
" )\n",
" text = resp.choices[0].message.content\n",
" usage = resp.usage\n",
" return {\n",
" \"raw_text\": text,\n",
" \"prompt_tokens\": getattr(usage, \"prompt_tokens\", None),\n",
" \"completion_tokens\": getattr(usage, \"completion_tokens\", None),\n",
" \"total_tokens\": getattr(usage, \"total_tokens\", None),\n",
" \"api_error\": None,\n",
" }\n",
" except Exception as e:\n",
" if attempt >= MAX_RETRIES:\n",
" return {\"raw_text\": \"\", \"prompt_tokens\": None, \"completion_tokens\": None, \"total_tokens\": None, \"api_error\": str(e)}\n",
" time.sleep(1.0 + attempt)\n",
"\n",
"def extract_json(text):\n",
" text = (text or \"\").strip()\n",
" if text.startswith(\"```\"):\n",
" text = re.sub(r\"^```(?:json)?\", \"\", text).strip()\n",
" text = re.sub(r\"```$\", \"\", text).strip()\n",
" try:\n",
" return json.loads(text), True\n",
" except Exception:\n",
" pass\n",
" m = re.search(r\"\\{.*\\}\", text, flags=re.S)\n",
" if m:\n",
" try:\n",
" return json.loads(m.group(0)), True\n",
" except Exception:\n",
" pass\n",
" return None, False\n",
"\n",
"def repair_json(raw_text):\n",
" schema = {\n",
" \"answer\": \"\",\n",
" \"decision\": \"allow|block|insufficient|limited|overclaim|unknown\",\n",
" \"used_evidence\": [],\n",
" \"boundary_notes\": [],\n",
" \"confidence\": \"high|medium|low\",\n",
" \"insufficient_evidence\": \"true|false\",\n",
" \"claim_ceiling_observed\": \"true|false\",\n",
" }\n",
" repair_prompt = \"\\n\".join([\n",
" \"Convert the following model response into valid JSON using exactly this schema:\",\n",
" json.dumps(schema, indent=2),\n",
" \"Do not change the meaning. Do not add evidence. Only repair formatting.\",\n",
" \"Original response:\",\n",
" raw_text or \"\",\n",
" ])\n",
" repaired = call_model(MODEL_SMALL, repair_prompt)\n",
" obj, ok = extract_json(repaired[\"raw_text\"])\n",
" return obj, ok\n",
"\n",
"def normalize_parsed(obj):\n",
" if not isinstance(obj, dict):\n",
" obj = {}\n",
" allowed = {\"allow\", \"block\", \"insufficient\", \"limited\", \"overclaim\", \"unknown\"}\n",
" decision = str(obj.get(\"decision\", \"unknown\")).lower().strip()\n",
" used = obj.get(\"used_evidence\", [])\n",
" notes = obj.get(\"boundary_notes\", [])\n",
" conf = str(obj.get(\"confidence\", \"low\")).lower().strip()\n",
" return {\n",
" \"answer\": str(obj.get(\"answer\", \"\")),\n",
" \"decision\": decision if decision in allowed else \"unknown\",\n",
" \"used_evidence\": used if isinstance(used, list) else [str(used)],\n",
" \"boundary_notes\": notes if isinstance(notes, list) else [str(notes)],\n",
" \"confidence\": conf if conf in {\"high\", \"medium\", \"low\"} else \"low\",\n",
" \"insufficient_evidence\": bool(obj.get(\"insufficient_evidence\", False)),\n",
" \"claim_ceiling_observed\": bool(obj.get(\"claim_ceiling_observed\", True)),\n",
" }\n",
"\n",
"def safe_token_estimate(text):\n",
" if tiktoken is None:\n",
" return max(1, len(str(text).split()))\n",
" try:\n",
" enc = tiktoken.get_encoding(\"cl100k_base\")\n",
" return len(enc.encode(str(text)))\n",
" except Exception:\n",
" return max(1, len(str(text).split()))\n",
"\n",
"def sentence_windows(text):\n",
" text = (text or \"\").lower()\n",
" parts = re.split(r\"(?<=[.!?;])\\s+|\\n+\", text)\n",
" return [p.strip() for p in parts if p.strip()]\n",
"\n",
"NEGATION_CUES = [\n",
" \"cannot\", \"can not\", \"can't\", \"do not\", \"does not\", \"not\", \"no \",\n",
" \"never\", \"must not\", \"should not\", \"insufficient\", \"not enough\",\n",
" \"not proven\", \"not established\", \"not confirmed\", \"absent\",\n",
" \"missing\", \"without\", \"lacks\", \"cannot claim\", \"cannot be described\",\n",
"]\n",
"AFFIRMATION_CUES = [\n",
" \"can claim\", \"may claim\", \"is production safe\", \"production safe is established\",\n",
" \"universal\", \"always\", \"fully proven\", \"guaranteed\", \"approved\",\n",
" \"confirmed\", \"promised\", \"allowed\", \"can be treated as\", \"is enough\",\n",
"]\n",
"\n",
"def term_is_negated(text, term):\n",
" term = str(term).lower()\n",
" for sent in sentence_windows(text):\n",
" if term in sent and any(cue in sent for cue in NEGATION_CUES):\n",
" return True\n",
" return False\n",
"\n",
"def term_is_affirmed(text, term):\n",
" term = str(term).lower()\n",
" for sent in sentence_windows(text):\n",
" if term in sent:\n",
" has_neg = any(cue in sent for cue in NEGATION_CUES)\n",
" has_aff = any(cue in sent for cue in AFFIRMATION_CUES)\n",
" if has_aff and not has_neg:\n",
" return True\n",
" if not has_neg and any(v in sent for v in [\" is \", \" are \", \" can \", \" may \", \" should \"]):\n",
" return True\n",
" return False\n",
"\n",
"def reason_alignment_score(case, parsed):\n",
" text = (parsed.get(\"answer\", \"\") + \" \" + \" \".join(parsed.get(\"used_evidence\", [])) + \" \" + \" \".join(parsed.get(\"boundary_notes\", []))).lower()\n",
" req = case.get(\"required_terms\", [])\n",
" if not req:\n",
" return 1.0\n",
" hits = sum(1 for t in req if str(t).lower() in text)\n",
" if hits >= min(2, len(req)):\n",
" return 1.0\n",
" if hits == 1:\n",
" return 0.75\n",
" if parsed.get(\"decision\") in {case.get(\"expected_decision\"), \"limited\", \"insufficient\", \"block\"} and parsed.get(\"used_evidence\"):\n",
" return 0.70\n",
" return 0.45\n",
"\n",
"def wrong_source_penalty(case, parsed):\n",
" text = (parsed.get(\"answer\", \"\") + \" \" + \" \".join(parsed.get(\"used_evidence\", [])) + \" \" + \" \".join(parsed.get(\"boundary_notes\", []))).lower()\n",
" terms = [str(t).lower() for t in case.get(\"wrong_source_terms\", [])]\n",
" source_use_patterns = [\n",
" \"based on {t}\", \"according to {t}\", \"using {t}\", \"use {t}\", \"uses {t}\",\n",
" \"rely on {t}\", \"relies on {t}\", \"{t} controls\", \"{t} is controlling\",\n",
" \"{t} is the authority\", \"{t} should be used\", \"{t} must be used\",\n",
" \"{t} governs\", \"{t} overrides\", \"{t} proves\", \"{t} confirms\",\n",
" ]\n",
" exclusion_patterns = [\n",
" \"not based on {t}\", \"not according to {t}\", \"do not use {t}\", \"does not use {t}\",\n",
" \"{t} does not control\", \"{t} doesn't control\", \"{t} is not controlling\",\n",
" \"{t} is not the authority\", \"{t} cannot override\", \"{t} should not be used\",\n",
" \"{t} is excluded\", \"{t} out of scope\", \"conflict with {t}\", \"over {t}\",\n",
" \"rather than {t}\", \"instead of {t}\", \"not {t}\",\n",
" ]\n",
" for t in terms:\n",
" for sent in sentence_windows(text):\n",
" if t not in sent:\n",
" continue\n",
" safe = any(pat.format(t=t) in sent for pat in exclusion_patterns)\n",
" if safe or term_is_negated(sent, t):\n",
" continue\n",
" if any(pat.format(t=t) in sent for pat in source_use_patterns):\n",
" return 0.15\n",
" if \"controls\" in sent and t in sent:\n",
" if any(safe_cue in sent for safe_cue in [\"conflict with\", \"over\", \"rather than\", \"instead of\"]):\n",
" continue\n",
" return 0.0\n",
"\n",
"def shadow_overclaim_penalty(case, parsed):\n",
" text = (parsed.get(\"answer\", \"\") + \" \" + \" \".join(parsed.get(\"boundary_notes\", []))).lower()\n",
" terms = case.get(\"shadow_overclaim_terms\", []) + [\"universal\", \"always\", \"fully proven\", \"production safe\", \"all cases\", \"guaranteed\"]\n",
" for t in terms:\n",
" t = str(t).lower()\n",
" if t in text:\n",
" if term_is_negated(text, t):\n",
" continue\n",
" if term_is_affirmed(text, t):\n",
" return 0.20\n",
" return 0.0\n",
"\n",
"def family_failure_text(case, answer_text, decision):\n",
" text = answer_text.lower()\n",
" if decision == case[\"wrong_decision\"] or decision == \"overclaim\":\n",
" return True\n",
" if any(x in text for x in [\n",
" \"allow\", \"allowed\", \"approved\", \"confirmed\", \"valid\", \"proven\", \"universal\", \"production safe\",\n",
" \"old archive\", \"prior case\", \"surface wording\", \"one favorable condition\", \"repeated approval\",\n",
" \"single favorable condition\", \"all required constraints treated as satisfied\",\n",
" \"universal superiority\", \"all task domains\", \"hype archive\", \"production guarantee\",\n",
" \"production safe\", \"all domains\", \"beats all models\", \"deployment ready\",\n",
" \"blind archive claims universal\", \"old repeated approval\", \"stale archive\", \"red archive flood\",\n",
" \"hashing note\", \"raw linkable\", \"export boundary ignored\", \"marketing page\", \"signed order ignored\",\n",
" \"local claim ceiling ignored\", \"scoped boundary ignored\", \"public production guarantee\",\n",
" \"long context archive\", \"long hype archive\",\n",
" \"manager approval alone\", \"required training missing\", \"treated as sufficient\",\n",
" \"user urgency\", \"destructive production action\", \"treated as allowed\",\n",
" \"scope expansion archive\", \"current boundary overridden\",\n",
" \"noncontrolling source\", \"noncontrolling confident source\", \"controlling source ignored\",\n",
" \"noncontrolling_source_treated_as_final_authority\",\n",
" \"old repeated approval treated as current local authority\",\n",
" \"local active rule ignored\", \"current boundary overridden\",\n",
" \"v6_fallback_noncontrolling_source_used_as_authority\",\n",
" \"v7_repro_margin_failure_signature\",\n",
" \"scope expansion archive\", \"current boundary overridden\",\n",
" \"signed order ignored\", \"raw linkable customer identifier\",\n",
" \"required training missing\", \"destructive production action\"\n",
" ]):\n",
" if case[\"wrong_decision\"] in {\"allow\", \"overclaim\"}:\n",
" return True\n",
" return False\n",
"\n",
"def f_flip_match(case, parsed, answer_text):\n",
" decision = parsed[\"decision\"]\n",
" text = answer_text.lower()\n",
" if decision != case[\"expected_decision\"] and decision in {\"allow\", \"block\", \"limited\", \"insufficient\"}:\n",
" return True\n",
" if any(x in text for x in [\"condition changed\", \"changed\", \"allow\", \"limited\", \"limited allow\", \"limited scope\", \"external audit present\", \"valid allow variant\", \"all required constraints now satisfied\", \"current state all constraints hold\", \"previous blocking condition removed\"]):\n",
" return True\n",
" return False\n",
"\n",
"def base_judge(case, arm, parsed):\n",
" decision = parsed[\"decision\"]\n",
" answer_text = (parsed.get(\"answer\", \"\") + \" \" + \" \".join(parsed.get(\"used_evidence\", [])) + \" \" + \" \".join(parsed.get(\"boundary_notes\", []))).lower()\n",
" expected = case[\"expected_decision\"]\n",
"\n",
" truth = 1.0 if decision == expected else 0.7 if (expected == \"limited\" and decision in {\"limited\", \"insufficient\"}) else 0.35\n",
" route = 1.0 if parsed[\"used_evidence\"] else 0.75\n",
" boundary = 1.0 if parsed[\"claim_ceiling_observed\"] else 0.5\n",
" evidence = 1.0 if parsed[\"used_evidence\"] else 0.6\n",
" claim = 1.0 if decision != \"overclaim\" else 0.25\n",
" contract = 1.0\n",
"\n",
" failure_match = None\n",
" if arm == \"D\":\n",
" failure_match = family_failure_text(case, answer_text, decision)\n",
" if failure_match:\n",
" truth = min(truth, 0.3)\n",
"\n",
" ablation_effect = None\n",
" if arm == \"E\":\n",
" missing_language = any(x in answer_text for x in [\"insufficient\", \"missing\", \"not enough\", \"cannot determine\", \"unknown\", \"lacks\"])\n",
" ablation_effect = decision in {\"unknown\", \"insufficient\"} or missing_language or decision != expected\n",
" if ablation_effect:\n",
" truth = min(truth, 0.45)\n",
" route = min(route, 0.65)\n",
" evidence = min(evidence, 0.55)\n",
" boundary = min(boundary, 0.65)\n",
"\n",
" f_missing = None\n",
" f_flip = None\n",
" if arm == \"F\":\n",
" if case[\"f_mode\"] == \"missing\":\n",
" f_missing = decision in {\"insufficient\", \"limited\", \"block\"} or parsed[\"insufficient_evidence\"]\n",
" if f_missing:\n",
" truth = max(truth, 0.9)\n",
" else:\n",
" f_flip = f_flip_match(case, parsed, answer_text)\n",
" if f_flip:\n",
" truth = max(truth, 0.9)\n",
"\n",
" base_score = sum([truth, route, boundary, evidence, claim, contract]) / 6.0\n",
" reason_score = reason_alignment_score(case, parsed)\n",
" wrong_penalty = wrong_source_penalty(case, parsed)\n",
" shadow_penalty = shadow_overclaim_penalty(case, parsed)\n",
"\n",
" if arm == \"C\" and base_score == 1.0 and wrong_penalty == 0.0 and shadow_penalty == 0.0:\n",
" reason_score = max(reason_score, 0.90)\n",
"\n",
" claim_safety_score = 1.0 - min(1.0, shadow_penalty / 0.20)\n",
" final_score = max(0.0, min(1.0, 0.70 * base_score + 0.20 * reason_score + 0.10 * claim_safety_score - wrong_penalty - shadow_penalty))\n",
"\n",
" return {\n",
" \"truth_score\": round(truth, 4),\n",
" \"route_score\": round(route, 4),\n",
" \"boundary_score\": round(boundary, 4),\n",
" \"evidence_score\": round(evidence, 4),\n",
" \"claim_ceiling_score\": round(claim, 4),\n",
" \"contract_score\": round(contract, 4),\n",
" \"base_score\": round(base_score, 4),\n",
" \"reason_alignment_score\": round(reason_score, 4),\n",
" \"claim_safety_score\": round(claim_safety_score, 4),\n",
" \"wrong_source_penalty\": round(wrong_penalty, 4),\n",
" \"shadow_overclaim_penalty\": round(shadow_penalty, 4),\n",
" \"score\": round(final_score, 4),\n",
" \"failure_signature_match\": failure_match,\n",
" \"ablation_effect_observed\": ablation_effect,\n",
" \"missing_stop_observed\": f_missing,\n",
" \"condition_flip_observed\": f_flip,\n",
" \"leakage_detected\": False,\n",
" }\n",
"\n",
"# Incremental checkpoint / resume runner.\n",
"# This section writes every completed output to JSONL immediately.\n",
"# If Colab disconnects but the runtime survives, rerun the cell and it will skip completed outputs.\n",
"# If Google Drive checkpoint is mounted, it can also resume after a runtime reset by reusing the latest incomplete RUN_ID.\n",
"\n",
"RAW_CHECKPOINT = OUT_DIR / \"checkpoint_raw_rows.jsonl\"\n",
"PARSED_CHECKPOINT = OUT_DIR / \"checkpoint_parsed_rows.jsonl\"\n",
"CASE_CHECKPOINT = OUT_DIR / \"checkpoint_case_verdict_rows.jsonl\"\n",
"PROGRESS_STATE = OUT_DIR / \"progress_state.json\"\n",
"\n",
"def _append_jsonl(path, record):\n",
" with open(path, \"a\", encoding=\"utf-8\") as f:\n",
" f.write(json.dumps(record, ensure_ascii=False) + \"\\n\")\n",
" f.flush()\n",
" try:\n",
" os.fsync(f.fileno())\n",
" except Exception:\n",
" pass\n",
"\n",
"def _read_jsonl(path):\n",
" rows = []\n",
" if not pathlib.Path(path).exists():\n",
" return rows\n",
" with open(path, \"r\", encoding=\"utf-8\") as f:\n",
" for line in f:\n",
" line = line.strip()\n",
" if not line:\n",
" continue\n",
" try:\n",
" rows.append(json.loads(line))\n",
" except Exception:\n",
" print(\"Skipping malformed checkpoint line in\", path, flush=True)\n",
" return rows\n",
"\n",
"raw_rows = _read_jsonl(RAW_CHECKPOINT)\n",
"parsed_rows = _read_jsonl(PARSED_CHECKPOINT)\n",
"case_verdict_rows = _read_jsonl(CASE_CHECKPOINT)\n",
"\n",
"completed_keys = set((r[\"case_id\"], r[\"arm\"]) for r in case_verdict_rows if \"case_id\" in r and \"arm\" in r)\n",
"TOTAL_OUTPUTS = len(CASES) * len(ARMS)\n",
"completed_outputs = len(completed_keys)\n",
"\n",
"print(\"=\" * 80, flush=True)\n",
"print(\"WFGY PP01 Final 320 v8 progress/resume runner\", flush=True)\n",
"print(\"Run ID:\", RUN_ID, flush=True)\n",
"print(\"Checkpoint directory:\", OUT_DIR, flush=True)\n",
"print(\"Total planned outputs:\", TOTAL_OUTPUTS, flush=True)\n",
"print(\"Already completed outputs:\", completed_outputs, flush=True)\n",
"print(\"Remaining outputs:\", TOTAL_OUTPUTS - completed_outputs, flush=True)\n",
"print(\"Progress print interval:\", PROGRESS_EVERY_N_OUTPUTS, \"outputs\", flush=True)\n",
"print(\"Checkpoint interval:\", CHECKPOINT_EVERY_N_OUTPUTS, \"outputs\", flush=True)\n",
"print(\"=\" * 80, flush=True)\n",
"\n",
"def _write_progress_state(extra=None):\n",
" state = {\n",
" \"run_id\": RUN_ID,\n",
" \"timestamp_utc\": datetime.now(timezone.utc).isoformat(),\n",
" \"total_outputs\": TOTAL_OUTPUTS,\n",
" \"completed_outputs\": len(completed_keys),\n",
" \"remaining_outputs\": TOTAL_OUTPUTS - len(completed_keys),\n",
" \"percent\": round(100.0 * len(completed_keys) / max(1, TOTAL_OUTPUTS), 3),\n",
" \"checkpoint_dir\": str(OUT_DIR),\n",
" }\n",
" if extra:\n",
" state.update(extra)\n",
" PROGRESS_STATE.write_text(json.dumps(state, indent=2, ensure_ascii=False), encoding=\"utf-8\")\n",
"\n",
"_write_progress_state({\"status\": \"started_or_resumed\"})\n",
"\n",
"last_print_completed = completed_outputs\n",
"\n",
"for case_index, case in enumerate(CASES, start=1):\n",
" for arm_spec in ARMS:\n",
" arm = arm_spec[\"arm\"]\n",
" key = (case[\"case_id\"], arm)\n",
" if key in completed_keys:\n",
" continue\n",
"\n",
" model = arm_spec[\"model\"]\n",
" prompt = build_prompt(case, arm_spec)\n",
"\n",
" started_at = datetime.now(timezone.utc).isoformat()\n",
" result = call_model(model, prompt)\n",
" raw_text = result[\"raw_text\"]\n",
"\n",
" parsed_obj, initial_ok = extract_json(raw_text)\n",
" repair_attempted = False\n",
" repair_ok = False\n",
" if not initial_ok:\n",
" repair_attempted = True\n",
" parsed_obj, repair_ok = repair_json(raw_text)\n",
" final_ok = initial_ok or repair_ok\n",
" parsed = normalize_parsed(parsed_obj if final_ok else {})\n",
"\n",
" prompt_tokens = result[\"prompt_tokens\"] if result[\"prompt_tokens\"] is not None else safe_token_estimate(prompt)\n",
" completion_tokens = result[\"completion_tokens\"] if result[\"completion_tokens\"] is not None else safe_token_estimate(raw_text)\n",
" total_tokens = result[\"total_tokens\"] if result[\"total_tokens\"] is not None else prompt_tokens + completion_tokens\n",
"\n",
" raw_record = {\n",
" \"run_id\": RUN_ID, \"case_id\": case[\"case_id\"], \"layer\": case[\"layer\"], \"family_id\": case[\"family_id\"],\n",
" \"case_role\": case[\"case_role\"], \"arm\": arm, \"model\": model, \"role\": arm_spec[\"role\"],\n",
" \"prompt\": prompt, \"raw_output\": raw_text, \"api_error\": result[\"api_error\"],\n",
" \"prompt_tokens\": prompt_tokens, \"completion_tokens\": completion_tokens, \"total_tokens\": total_tokens,\n",
" \"started_at_utc\": started_at, \"finished_at_utc\": datetime.now(timezone.utc).isoformat(),\n",
" }\n",
"\n",
" judge = base_judge(case, arm, parsed)\n",
"\n",
" parsed_record = {\n",
" \"run_id\": RUN_ID, \"case_id\": case[\"case_id\"], \"layer\": case[\"layer\"], \"family_id\": case[\"family_id\"], \"arm\": arm,\n",
" **parsed, \"initial_parse_pass\": initial_ok, \"repair_attempted\": repair_attempted,\n",
" \"repair_parse_pass\": repair_ok, \"final_parse_pass\": final_ok,\n",
" }\n",
"\n",
" case_record = {\n",
" \"run_id\": RUN_ID, \"case_id\": case[\"case_id\"], \"layer\": case[\"layer\"], \"family_id\": case[\"family_id\"],\n",
" \"family_name\": case[\"family_name\"], \"case_role\": case[\"case_role\"], \"arm\": arm,\n",
" \"model\": model, \"expected_decision\": case[\"expected_decision\"], \"observed_decision\": parsed[\"decision\"],\n",
" \"f_mode\": case[\"f_mode\"], \"final_parse_pass\": final_ok,\n",
" \"prompt_tokens\": prompt_tokens, \"completion_tokens\": completion_tokens, \"total_tokens\": total_tokens,\n",
" **judge,\n",
" }\n",
"\n",
" # Write checkpoint first, then update memory state.\n",
" _append_jsonl(RAW_CHECKPOINT, raw_record)\n",
" _append_jsonl(PARSED_CHECKPOINT, parsed_record)\n",
" _append_jsonl(CASE_CHECKPOINT, case_record)\n",
"\n",
" raw_rows.append(raw_record)\n",
" parsed_rows.append(parsed_record)\n",
" case_verdict_rows.append(case_record)\n",
" completed_keys.add(key)\n",
"\n",
" completed_now = len(completed_keys)\n",
" should_print = (\n",
" completed_now == TOTAL_OUTPUTS\n",
" or completed_now - last_print_completed >= PROGRESS_EVERY_N_OUTPUTS\n",
" or arm == \"F\"\n",
" )\n",
"\n",
" if should_print:\n",
" pct = 100.0 * completed_now / max(1, TOTAL_OUTPUTS)\n",
" d_match = case_record.get(\"failure_signature_match\", \"\")\n",
" print(\n",
" f\"[{datetime.now(timezone.utc).strftime('%H:%M:%S')}] \"\n",
" f\"{completed_now}/{TOTAL_OUTPUTS} ({pct:.1f}%) \"\n",
" f\"case={case['case_id']} layer={case['layer']} arm={arm} \"\n",
" f\"score={case_record.get('score')} Dmatch={d_match} \"\n",
" f\"tokens={total_tokens}\",\n",
" flush=True\n",
" )\n",
" last_print_completed = completed_now\n",
"\n",
" if completed_now % CHECKPOINT_EVERY_N_OUTPUTS == 0 or completed_now == TOTAL_OUTPUTS:\n",
" _write_progress_state({\n",
" \"status\": \"running\" if completed_now < TOTAL_OUTPUTS else \"model_outputs_complete\",\n",
" \"last_case_id\": case[\"case_id\"],\n",
" \"last_arm\": arm,\n",
" \"last_score\": case_record.get(\"score\"),\n",
" \"last_total_tokens\": total_tokens,\n",
" })\n",
"\n",
" time.sleep(SLEEP_SECONDS)\n",
"\n",
"_write_progress_state({\"status\": \"all_model_outputs_checkpointed\"})\n",
"print(\"=\" * 80, flush=True)\n",
"print(\"Model output collection complete:\", len(completed_keys), \"/\", TOTAL_OUTPUTS, flush=True)\n",
"print(\"Checkpoint files:\", flush=True)\n",
"print(\" -\", RAW_CHECKPOINT, flush=True)\n",
"print(\" -\", PARSED_CHECKPOINT, flush=True)\n",
"print(\" -\", CASE_CHECKPOINT, flush=True)\n",
"print(\"=\" * 80, flush=True)\n",
"\n",
"raw_df = pd.DataFrame(raw_rows)\n",
"parsed_df = pd.DataFrame(parsed_rows)\n",
"case_df = pd.DataFrame(case_verdict_rows)\n",
"\n",
"raw_df.to_json(OUT_DIR / \"02_OSK_FINAL_320_RAW_OUTPUTS.jsonl\", orient=\"records\", lines=True, force_ascii=False)\n",
"parsed_df.to_json(OUT_DIR / \"03_OSK_FINAL_320_PARSED_OUTPUTS.json\", orient=\"records\", force_ascii=False, indent=2)\n",
"case_df.to_csv(OUT_DIR / \"05_OSK_FINAL_320_CASE_LEVEL_VERDICTS.csv\", index=False)\n",
"\n",
"arm_summary = case_df.groupby(\"arm\").agg(\n",
" score_mean=(\"score\", \"mean\"),\n",
" score_min=(\"score\", \"min\"),\n",
" reason_alignment_mean=(\"reason_alignment_score\", \"mean\"),\n",
" wrong_source_penalty_sum=(\"wrong_source_penalty\", \"sum\"),\n",
" shadow_overclaim_penalty_sum=(\"shadow_overclaim_penalty\", \"sum\"),\n",
" prompt_tokens=(\"prompt_tokens\", \"sum\"),\n",
" completion_tokens=(\"completion_tokens\", \"sum\"),\n",
" total_tokens=(\"total_tokens\", \"sum\"),\n",
" parse_pass_rate=(\"final_parse_pass\", \"mean\"),\n",
" leakage_count=(\"leakage_detected\", \"sum\"),\n",
").reset_index()\n",
"\n",
"token_df = arm_summary[[\"arm\", \"prompt_tokens\", \"completion_tokens\", \"total_tokens\"]].copy()\n",
"\n",
"def sf(df, arm, col):\n",
" s = df.loc[df.arm == arm, col]\n",
" return float(s.iloc[0]) if len(s) else 0.0\n",
"\n",
"A_total = sf(token_df, \"A\", \"total_tokens\")\n",
"A_prompt = sf(token_df, \"A\", \"prompt_tokens\")\n",
"C_total = sf(token_df, \"C\", \"total_tokens\")\n",
"C_prompt = sf(token_df, \"C\", \"prompt_tokens\")\n",
"\n",
"family_rows = []\n",
"for family_id, fam_cases in case_df.groupby(\"family_id\"):\n",
" c_scores = fam_cases[fam_cases.arm == \"C\"][\"score\"]\n",
" d_matches = fam_cases[fam_cases.arm == \"D\"][\"failure_signature_match\"].dropna()\n",
" family_pass = bool(c_scores.mean() >= 0.82 and (d_matches.mean() if len(d_matches) else 0) >= 0.5)\n",
" family_collapse = bool(c_scores.mean() < 0.65)\n",
" family_rows.append({\n",
" \"family_id\": family_id,\n",
" \"C_score_mean\": round(float(c_scores.mean()), 4),\n",
" \"D_match_rate\": round(float(d_matches.mean()), 4) if len(d_matches) else None,\n",
" \"family_pass\": family_pass,\n",
" \"family_collapse\": family_collapse,\n",
" })\n",
"family_df = pd.DataFrame(family_rows)\n",
"\n",
"layer_rows = []\n",
"for layer, layer_cases in case_df.groupby(\"layer\"):\n",
" c_scores = layer_cases[layer_cases.arm == \"C\"][\"score\"]\n",
" d_matches = layer_cases[layer_cases.arm == \"D\"][\"failure_signature_match\"].dropna()\n",
" layer_pass = bool(c_scores.mean() >= 0.95 and (d_matches.mean() if len(d_matches) else 0) >= 0.98)\n",
" layer_rows.append({\n",
" \"layer\": layer,\n",
" \"case_count\": int(layer_cases[\"case_id\"].nunique()),\n",
" \"C_score_mean\": round(float(c_scores.mean()), 4),\n",
" \"D_match_rate\": round(float(d_matches.mean()), 4) if len(d_matches) else None,\n",
" \"layer_pass\": layer_pass,\n",
" })\n",
"layer_df = pd.DataFrame(layer_rows)\n",
"\n",
"d_audit = case_df[case_df.arm == \"D\"][[\"run_id\", \"case_id\", \"layer\", \"family_id\", \"expected_decision\", \"observed_decision\", \"failure_signature_match\", \"score\", \"reason_alignment_score\"]].copy()\n",
"c_scores_by_case = case_df[case_df.arm == \"C\"].set_index(\"case_id\")[\"score\"].to_dict()\n",
"e_audit = case_df[case_df.arm == \"E\"][[\"run_id\", \"case_id\", \"layer\", \"family_id\", \"expected_decision\", \"observed_decision\", \"ablation_effect_observed\", \"score\", \"reason_alignment_score\"]].copy()\n",
"e_audit[\"C_score\"] = e_audit[\"case_id\"].map(c_scores_by_case)\n",
"e_audit[\"C_minus_E_score_delta\"] = e_audit[\"C_score\"] - e_audit[\"score\"]\n",
"f_audit = case_df[case_df.arm == \"F\"][[\"run_id\", \"case_id\", \"layer\", \"family_id\", \"f_mode\", \"expected_decision\", \"observed_decision\", \"missing_stop_observed\", \"condition_flip_observed\", \"score\", \"reason_alignment_score\"]].copy()\n",
"leakage_audit = case_df[[\"run_id\", \"case_id\", \"layer\", \"family_id\", \"arm\", \"leakage_detected\"]].copy()\n",
"scorer_audit = case_df[[\"run_id\", \"case_id\", \"layer\", \"family_id\", \"arm\", \"base_score\", \"reason_alignment_score\", \"claim_safety_score\", \"wrong_source_penalty\", \"shadow_overclaim_penalty\", \"score\"]].copy()\n",
"\n",
"expected_outputs = len(CASES) * len(ARMS)\n",
"actual_outputs = len(case_df)\n",
"parse_pass_rate = float(case_df[\"final_parse_pass\"].mean()) if len(case_df) else 0.0\n",
"leakage_count = int(case_df[\"leakage_detected\"].sum()) if len(case_df) else 999\n",
"\n",
"C_score = float(case_df[case_df.arm == \"C\"][\"score\"].mean()) if len(case_df[case_df.arm == \"C\"]) else 0.0\n",
"B_score = float(case_df[case_df.arm == \"B\"][\"score\"].mean()) if len(case_df[case_df.arm == \"B\"]) else 0.0\n",
"C_retention_vs_B = C_score / B_score if B_score else None\n",
"C_raw_total_ratio_vs_A = C_total / A_total if A_total else None\n",
"C_input_ratio_vs_A = C_prompt / A_prompt if A_prompt else None\n",
"\n",
"D_match_rate = float(d_audit[\"failure_signature_match\"].mean()) if len(d_audit) else 0.0\n",
"E_delta_mean = float(e_audit[\"C_minus_E_score_delta\"].mean()) if len(e_audit) else 0.0\n",
"\n",
"missing_vals, flip_vals = [], []\n",
"for _, row in f_audit.iterrows():\n",
" if row[\"f_mode\"] == \"missing\":\n",
" missing_vals.append(bool(row[\"missing_stop_observed\"]))\n",
" else:\n",
" flip_vals.append(bool(row[\"condition_flip_observed\"]))\n",
"F_missing_rate = sum(missing_vals) / len(missing_vals) if missing_vals else None\n",
"F_flip_rate = sum(flip_vals) / len(flip_vals) if flip_vals else None\n",
"\n",
"family_collapse_count = int(family_df[\"family_collapse\"].sum()) if len(family_df) else 999\n",
"reason_alignment_mean = float(case_df[case_df.arm == \"C\"][\"reason_alignment_score\"].mean()) if len(case_df[case_df.arm == \"C\"]) else 0.0\n",
"C_only = case_df[case_df.arm == \"C\"]\n",
"shadow_overclaim_count = int((C_only[\"shadow_overclaim_penalty\"] > 0).sum())\n",
"wrong_source_critical_count = int((C_only[\"wrong_source_penalty\"] > 0).sum())\n",
"red_team_pass_rate = float(layer_df.loc[layer_df.layer == \"red_team_stress\", \"layer_pass\"].iloc[0]) if len(layer_df.loc[layer_df.layer == \"red_team_stress\"]) else 0.0\n",
"blind_holdout_pass_rate = float(layer_df.loc[layer_df.layer == \"blind_holdout\", \"layer_pass\"].iloc[0]) if len(layer_df.loc[layer_df.layer == \"blind_holdout\"]) else 0.0\n",
"core_continuity_pass_rate = float(layer_df.loc[layer_df.layer == \"core_continuity\", \"layer_pass\"].iloc[0]) if len(layer_df.loc[layer_df.layer == \"core_continuity\"]) else 0.0\n",
"\n",
"hard_gates_pass = expected_outputs == actual_outputs and parse_pass_rate == 1.0 and leakage_count == 0\n",
"\n",
"final_pass = (\n",
" hard_gates_pass and\n",
" (C_retention_vs_B is not None and C_retention_vs_B >= 1.00) and\n",
" (C_raw_total_ratio_vs_A is not None and C_raw_total_ratio_vs_A <= 0.45) and\n",
" D_match_rate >= 0.98 and\n",
" E_delta_mean >= 0.17 and\n",
" (F_missing_rate is not None and F_missing_rate == 1.0) and\n",
" (F_flip_rate is not None and F_flip_rate == 1.0) and\n",
" reason_alignment_mean >= 0.95 and\n",
" shadow_overclaim_count == 0 and\n",
" wrong_source_critical_count == 0 and\n",
" family_collapse_count == 0 and\n",
" bool(layer_df[\"layer_pass\"].all())\n",
")\n",
"\n",
"if not hard_gates_pass:\n",
" global_status = \"OSK_MVP_FINAL_HALT\"\n",
"elif final_pass:\n",
" global_status = \"OSK_MVP_FINAL_PASS\"\n",
"else:\n",
" global_status = \"OSK_MVP_FINAL_NEAR_PASS\"\n",
"\n",
"global_verdict = {\n",
" \"run_id\": RUN_ID,\n",
" \"stage\": \"PP01_T4_ONE_SHOT_KILLER_MVP_FINAL_320_CASES_V8_PROGRESS_RESUME\",\n",
" \"case_count\": len(CASES),\n",
" \"arm_count\": len(ARMS),\n",
" \"expected_outputs\": expected_outputs,\n",
" \"actual_outputs\": actual_outputs,\n",
" \"parse_pass_rate\": parse_pass_rate,\n",
" \"leakage_count\": leakage_count,\n",
" \"C_score_mean\": C_score,\n",
" \"B_score_mean\": B_score,\n",
" \"C_retention_vs_B\": C_retention_vs_B,\n",
" \"C_raw_total_ratio_vs_A\": C_raw_total_ratio_vs_A,\n",
" \"C_input_ratio_vs_A\": C_input_ratio_vs_A,\n",
" \"D_failure_signature_match_rate\": D_match_rate,\n",
" \"E_ablation_delta_mean\": E_delta_mean,\n",
" \"F_missing_stop_rate\": F_missing_rate,\n",
" \"F_condition_flip_rate\": F_flip_rate,\n",
" \"reason_alignment_mean\": reason_alignment_mean,\n",
" \"shadow_overclaim_count\": shadow_overclaim_count,\n",
" \"wrong_source_critical_count\": wrong_source_critical_count,\n",
" \"family_collapse_count\": family_collapse_count,\n",
" \"core_continuity_layer_pass\": bool(layer_df.loc[layer_df.layer == \"core_continuity\", \"layer_pass\"].iloc[0]) if len(layer_df.loc[layer_df.layer == \"core_continuity\"]) else False,\n",
" \"blind_holdout_layer_pass\": bool(layer_df.loc[layer_df.layer == \"blind_holdout\", \"layer_pass\"].iloc[0]) if len(layer_df.loc[layer_df.layer == \"blind_holdout\"]) else False,\n",
" \"red_team_layer_pass\": bool(layer_df.loc[layer_df.layer == \"red_team_stress\", \"layer_pass\"].iloc[0]) if len(layer_df.loc[layer_df.layer == \"red_team_stress\"]) else False,\n",
" \"hard_gates_pass\": hard_gates_pass,\n",
" \"final_pass\": final_pass,\n",
" \"global_status\": global_status,\n",
" \"recommended_next_action\": \"Proceed to final 320-case OSK MVP notebook if pilot passes; otherwise analyze weak layers and blockers.\",\n",
"}\n",
"\n",
"# Write outputs\n",
"arm_summary.to_csv(OUT_DIR / \"09_OSK_FINAL_320_ARM_SUMMARY.csv\", index=False)\n",
"token_df.to_csv(OUT_DIR / \"07_OSK_FINAL_320_TOKEN_ACCOUNTING.csv\", index=False)\n",
"family_df.to_csv(OUT_DIR / \"06_OSK_FINAL_320_FAMILY_LEVEL_VERDICTS.csv\", index=False)\n",
"layer_df.to_csv(OUT_DIR / \"06B_OSK_FINAL_320_LAYER_LEVEL_VERDICTS.csv\", index=False)\n",
"d_audit.to_csv(OUT_DIR / \"08_OSK_FINAL_320_D_FAILURE_SIGNATURE_AUDIT.csv\", index=False)\n",
"e_audit.to_csv(OUT_DIR / \"09_OSK_FINAL_320_E_ABLATION_AUDIT.csv\", index=False)\n",
"f_audit.to_csv(OUT_DIR / \"10_OSK_FINAL_320_F_MISSING_FLIP_AUDIT.csv\", index=False)\n",
"leakage_audit.to_csv(OUT_DIR / \"11_OSK_FINAL_320_LEAKAGE_AUDIT.csv\", index=False)\n",
"scorer_audit.to_csv(OUT_DIR / \"16_OSK_FINAL_320_FINAL_SCORER_AUDIT.csv\", index=False)\n",
"\n",
"with open(OUT_DIR / \"12_OSK_FINAL_320_GLOBAL_VERDICT.json\", \"w\") as f:\n",
" json.dump(global_verdict, f, indent=2)\n",
"\n",
"case_contract_rows = []\n",
"for c in CASES:\n",
" case_contract_rows.append({k: c[k] for k in [\"case_id\", \"layer\", \"family_id\", \"family_name\", \"case_role\", \"f_mode\", \"task_prompt\", \"expected_decision\", \"wrong_decision\", \"expected_failure_signature_D\", \"expected_ablation_effect_E\", \"expected_F_behavior\"]})\n",
"pd.DataFrame(case_contract_rows).to_csv(OUT_DIR / \"01_OSK_FINAL_320_CASE_CONTRACT.csv\", index=False)\n",
"\n",
"summary = f\"\"\"# PP01 T4 One Shot Killer MVP Pilot v2 Local D/F Patch Summary\n",
"\n",
"Run ID: {RUN_ID}\n",
"Global status: {global_verdict[\"global_status\"]}\n",
"\n",
"Expected outputs: {global_verdict[\"expected_outputs\"]}\n",
"Actual outputs: {global_verdict[\"actual_outputs\"]}\n",
"Parse pass rate: {global_verdict[\"parse_pass_rate\"]}\n",
"Leakage count: {global_verdict[\"leakage_count\"]}\n",
"C retention vs B: {global_verdict[\"C_retention_vs_B\"]}\n",
"C raw total ratio vs A: {global_verdict[\"C_raw_total_ratio_vs_A\"]}\n",
"D failure signature match rate: {global_verdict[\"D_failure_signature_match_rate\"]}\n",
"E ablation delta mean: {global_verdict[\"E_ablation_delta_mean\"]}\n",
"F missing stop rate: {global_verdict[\"F_missing_stop_rate\"]}\n",
"F condition flip rate: {global_verdict[\"F_condition_flip_rate\"]}\n",
"Reason alignment mean: {global_verdict[\"reason_alignment_mean\"]}\n",
"Shadow overclaim count: {global_verdict[\"shadow_overclaim_count\"]}\n",
"Wrong source critical count: {global_verdict[\"wrong_source_critical_count\"]}\n",
"Family collapse count: {global_verdict[\"family_collapse_count\"]}\n",
"\n",
"Recommended next action: {global_verdict[\"recommended_next_action\"]}\n",
"\"\"\"\n",
"(OUT_DIR / \"00_OSK_FINAL_320_REAL_SUMMARY.md\").write_text(summary)\n",
"\n",
"scope = \"\"\"# Experiment Scope\n",
"\n",
"This is the 320-case final One Shot Killer MVP run for PP01 T4 One Shot Killer Gate MVP.\n",
"\n",
"Included:\n",
"- 160 core continuity cases\n",
"- 80 blind holdout cases\n",
"- 80 red team stress cases\n",
"- 6 arms\n",
"- 1920 expected outputs\n",
"\n",
"Excluded:\n",
"- larger 640-case ultimate run\n",
"- coding benchmarks\n",
"- formal math theorem proving\n",
"- creative generation\n",
"- multimodal grounding\n",
"- OCR, chart, and visual reasoning\n",
"- autonomous agent tool use\n",
"- universal small model superiority claims\n",
"\"\"\"\n",
"(OUT_DIR / \"00_SCOPE.md\").write_text(scope)\n",
"\n",
"blackfan = f\"\"\"# Blackfan Audit\n",
"\n",
"Verdict: {global_verdict[\"global_status\"]}\n",
"\n",
"Checks:\n",
"- Expected outputs equal actual outputs: {expected_outputs == actual_outputs}\n",
"- Final parse pass rate is 1.0: {parse_pass_rate == 1.0}\n",
"- Leakage count is zero: {leakage_count == 0}\n",
"- 320 final cases present: {len(CASES) == 80}\n",
"- 1920 expected outputs present: {expected_outputs == 480}\n",
"- Core continuity layer present: true\n",
"- Blind holdout layer present: true\n",
"- Red team stress layer present: true\n",
"- Final Scorer v5 inherited: true\\n- Runtime fix: theme_map defined before red-team final case generation: true\\n- v3 D local patch applied to six v2 D misses: true\\n- v4 stability patch applied to F05_C05 and F09_C06 latent D non-matches: true\\n- v5 reproducibility hardening applied to D-arm fragile classes: true\\n- v6 public reproducibility intro added: true\\n- v6 universal D fallback hardening applied: true\\n- v7 reproducibility margin hardening applied to D-arm packets: true\\n- v8 progress logging, checkpointing, and resume support enabled: true\\n- OSK Pilot v2 local D/F patch applied only to F12_BL, F13_BL, F12_R03, F12_R01: true\n",
"- Sentence-level source-use scorer inherited: true\n",
"- Negation-aware shadow overclaim inherited: true\n",
"- D blind corruption without CORRUPT labels: true\n",
"- E ablation packets remove core fields: true\n",
"- D/E/F remain separated by arm: true\n",
"- PP02 domains are not claimed: true\n",
"- Universal small model superiority is not claimed: true\n",
"\"\"\"\n",
"(OUT_DIR / \"13_OSK_FINAL_320_BLACKFAN_AUDIT.md\").write_text(blackfan)\n",
"\n",
"public_safe = \"\"\"# Public Safe Summary Draft\n",
"\n",
"An internal 320-case final was run for the PP01 T4 One Shot Killer Gate MVP.\n",
"\n",
"This pilot checks whether the 320-case design is safe to scale to the full final run.\n",
"\n",
"It includes core continuity, blind holdout, and red team stress layers. It does not disclose internal packet construction beyond this research artifact and should not be presented as a universal all-domain proof.\n",
"\"\"\"\n",
"(OUT_DIR / \"14_PUBLIC_SAFE_SUMMARY_DRAFT.md\").write_text(public_safe)\n",
"\n",
"manifest = {}\n",
"for p in sorted(OUT_DIR.iterdir()):\n",
" if p.is_file():\n",
" manifest[p.name] = hashlib.sha256(p.read_bytes()).hexdigest()\n",
"with open(OUT_DIR / \"15_MANIFEST_SHA256.json\", \"w\") as f:\n",
" json.dump(manifest, f, indent=2)\n",
"\n",
"zip_name = pathlib.Path(f\"/content/{RUN_ID}.zip\")\n",
"with zipfile.ZipFile(zip_name, \"w\", zipfile.ZIP_DEFLATED) as z:\n",
" for p in sorted(OUT_DIR.iterdir()):\n",
" z.write(p, arcname=f\"{RUN_ID}_OUTPUT/{p.name}\")\n",
"\n",
"print(json.dumps(global_verdict, indent=2))\n",
"print(\"Output dir:\", OUT_DIR)\n",
"print(\"Zip package:\", zip_name)\n",
"FINAL_DONE = OUT_DIR / \"final_done.json\"\n",
"FINAL_DONE.write_text(json.dumps({\"run_id\": RUN_ID, \"status\": global_verdict[\"global_status\"], \"timestamp_utc\": datetime.now(timezone.utc).isoformat()}, indent=2), encoding=\"utf-8\")\n",
"print(\"Download will start now.\")\n",
"\n",
"try:\n",
" from google.colab import files\n",
" files.download(str(zip_name))\n",
" print(\"Auto-download triggered:\", zip_name)\n",
"except Exception as e:\n",
" print(\"Auto-download failed:\", repr(e))\n",
" print(\"Manual download path:\", zip_name)\n"
]
},
{
"cell_type": "markdown",
"id": "8470b910",
"metadata": {},
"source": [
"---\n",
"\n",
"# Final Polaris Protocol report\n",
"\n",
"This final section generates a compact report for **PP01 — OneShotKiller 320 Main Cost Battlefield**.\n",
"\n",
"It produces:\n",
"- `polaris_report_PP01/executive_report.md`\n",
"- `polaris_report_PP01/metrics_summary.csv`\n",
"- `polaris_report_PP01/chart_*.png`\n",
"\n",
"The charts are intentionally simple. They are meant to make the experiment result easy to inspect, not to expand the claim boundary.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "6e9f56b5",
"metadata": {},
"outputs": [],
"source": [
"# Final Polaris Protocol report generator\n",
"# This cell uses the published result summary as a stable reporting fallback.\n",
"# If you adapt the notebook and compute fresh metrics, update POLARIS_PUBLISHED_METRICS before running this cell.\n",
"\n",
"from pathlib import Path\n",
"import json, csv, math, re\n",
"import pandas as pd\n",
"import matplotlib.pyplot as plt\n",
"\n",
"POLARIS_REPORT_EXPERIMENT_ID = \"PP01\"\n",
"POLARIS_REPORT_TITLE = \"PP01 — OneShotKiller 320 Main Cost Battlefield\"\n",
"POLARIS_REPORT_REPO = \"https://github.com/onestardao/WFGY\"\n",
"POLARIS_REPORT_PURPOSE = \"Tests whether topology-first task structure can reduce repeated long-context token workload while preserving inspectable quality and boundary behavior.\"\n",
"POLARIS_REPORT_SPIRIT = \"Main cost battlefield. This is where Polaris Protocol asks whether task topology can beat raw-context drag under scoped gates.\"\n",
"POLARIS_REPORT_CLAIM_BOUNDARY = \"Scoped cost / quality / boundary evidence only. Not universal token-saving proof, not invoice proof, not third-party certification.\"\n",
"POLARIS_PUBLISHED_SUMMARY = [\n",
" \"320 cases; 6 arms; 1920 expected outputs and 1920 actual outputs.\",\n",
" \"Structured C arm total token ratio vs A: 0.1656.\",\n",
" \"Structured C arm input token ratio vs A: 0.1242.\",\n",
" \"Overclaim count: 0; critical wrong-source count: 0; family collapse count: 0.\"\n",
"]\n",
"POLARIS_PUBLISHED_METRICS = {\n",
" \"C total token ratio vs A\": 0.1655887221,\n",
" \"C input token ratio vs A\": 0.1241701982,\n",
" \"B score mean\": 0.967265,\n",
" \"C score mean\": 0.988465,\n",
" \"Leakage count\": 0,\n",
" \"Shadow overclaim count\": 0,\n",
" \"Critical wrong-source count\": 0,\n",
" \"Family collapse count\": 0\n",
"}\n",
"POLARIS_CHART_SPECS = [\n",
" {\n",
" \"title\": \"Token ratio vs A\",\n",
" \"ylabel\": \"Ratio\",\n",
" \"labels\": [\n",
" \"A baseline\",\n",
" \"C structured\"\n",
" ],\n",
" \"values\": [\n",
" 1.0,\n",
" 0.1655887221\n",
" ]\n",
" },\n",
" {\n",
" \"title\": \"Score mean\",\n",
" \"ylabel\": \"Score\",\n",
" \"labels\": [\n",
" \"B\",\n",
" \"C structured\"\n",
" ],\n",
" \"values\": [\n",
" 0.967265,\n",
" 0.988465\n",
" ]\n",
" },\n",
" {\n",
" \"title\": \"Safety critical counts\",\n",
" \"ylabel\": \"Count\",\n",
" \"labels\": [\n",
" \"Leakage\",\n",
" \"Overclaim\",\n",
" \"Wrong-source\",\n",
" \"Family collapse\"\n",
" ],\n",
" \"values\": [\n",
" 0,\n",
" 0,\n",
" 0,\n",
" 0\n",
" ]\n",
" }\n",
"]\n",
"\n",
"report_dir = Path(f\"polaris_report_{POLARIS_REPORT_EXPERIMENT_ID}\")\n",
"report_dir.mkdir(exist_ok=True)\n",
"\n",
"# Save metrics table.\n",
"metrics_df = pd.DataFrame([\n",
" {\"metric\": key, \"value\": value}\n",
" for key, value in POLARIS_PUBLISHED_METRICS.items()\n",
"])\n",
"metrics_path = report_dir / \"metrics_summary.csv\"\n",
"metrics_df.to_csv(metrics_path, index=False)\n",
"\n",
"# Generate compact charts.\n",
"chart_paths = []\n",
"def _safe_chart_name(title):\n",
" return re.sub(r\"[^a-z0-9_\\\\-]+\", \"_\", title.lower().replace(\" \", \"_\")).strip(\"_\")\n",
"\n",
"for idx, spec in enumerate(POLARIS_CHART_SPECS, start=1):\n",
" labels = spec[\"labels\"]\n",
" values = spec[\"values\"]\n",
" plt.figure(figsize=(max(6, len(labels) * 1.2), 4))\n",
" bars = plt.bar(labels, values)\n",
" plt.title(spec[\"title\"])\n",
" plt.ylabel(spec.get(\"ylabel\", \"Value\"))\n",
" plt.xticks(rotation=25, ha=\"right\")\n",
"\n",
" numeric_values = [float(v) for v in values if isinstance(v, (int, float))]\n",
" max_value = max(numeric_values) if numeric_values else 0.0\n",
" min_value = min(numeric_values) if numeric_values else 0.0\n",
"\n",
" if max_value == 0 and min_value == 0:\n",
" # Make zero-count success charts visually readable instead of looking blank.\n",
" plt.ylim(0, 1)\n",
" for bar, value in zip(bars, values):\n",
" x = bar.get_x() + bar.get_width() / 2\n",
" plt.text(x, 0.05, str(value), ha=\"center\", va=\"bottom\", fontsize=10)\n",
" plt.text(\n",
" 0.5, 0.88,\n",
" \"All tracked counts are 0\",\n",
" transform=plt.gca().transAxes,\n",
" ha=\"center\",\n",
" va=\"center\",\n",
" fontsize=11,\n",
" )\n",
" else:\n",
" upper = max_value * 1.18 if max_value > 0 else 1\n",
" if min_value >= 0:\n",
" plt.ylim(0, upper)\n",
" for bar, value in zip(bars, values):\n",
" x = bar.get_x() + bar.get_width() / 2\n",
" y = bar.get_height()\n",
" label = f\"{value:.4g}\" if isinstance(value, float) else str(value)\n",
" plt.text(x, y + (upper * 0.02 if max_value > 0 else 0.03), label, ha=\"center\", va=\"bottom\", fontsize=9)\n",
"\n",
" plt.tight_layout()\n",
" chart_path = report_dir / f\"chart_{idx:02d}_{_safe_chart_name(spec['title'])}.png\"\n",
" plt.savefig(chart_path, dpi=180, bbox_inches=\"tight\")\n",
" plt.show()\n",
" chart_paths.append(chart_path)\n",
"\n",
"# Save executive markdown report.\n",
"summary_lines = \"\\n\".join([f\"- {item}\" for item in POLARIS_PUBLISHED_SUMMARY])\n",
"chart_lines = \"\\n\".join([f\"- {p.name}\" for p in chart_paths])\n",
"\n",
"report_md = f\"\"\"# {POLARIS_REPORT_TITLE}\n",
"\n",
"**Repository:** {POLARIS_REPORT_REPO} \n",
"**Experiment ID:** {POLARIS_REPORT_EXPERIMENT_ID}\n",
"\n",
"## Experiment spirit\n",
"\n",
"{POLARIS_REPORT_SPIRIT}\n",
"\n",
"## What this tests\n",
"\n",
"{POLARIS_REPORT_PURPOSE}\n",
"\n",
"## Published result summary\n",
"\n",
"{summary_lines}\n",
"\n",
"## Generated files\n",
"\n",
"- metrics_summary.csv\n",
"{chart_lines}\n",
"\n",
"## Claim boundary\n",
"\n",
"{POLARIS_REPORT_CLAIM_BOUNDARY}\n",
"\n",
"This report is designed for reproduction / inspection. It does not convert scoped evidence into universal proof.\n",
"\"\"\"\n",
"\n",
"report_path = report_dir / \"executive_report.md\"\n",
"report_path.write_text(report_md, encoding=\"utf-8\")\n",
"\n",
"print(\"Polaris Protocol report generated:\")\n",
"print(f\"- {report_path}\")\n",
"print(f\"- {metrics_path}\")\n",
"for path in chart_paths:\n",
" print(f\"- {path}\")\n"
]
}
],
"metadata": {
"colab": {
"provenance": []
},
"kernelspec": {
"display_name": "Python 3",
"name": "python3"
},
"language_info": {
"name": "python"
},
"polaris_protocol": {
"claim_boundary": "Scoped cost / quality / boundary evidence only. Not universal token-saving proof, not invoice proof, not third-party certification.",
"experiment_id": "PP01",
"experiment_title": "PP01 — OneShotKiller 320 Main Cost Battlefield",
"mvp_official_colab": true,
"repo": "https://github.com/onestardao/WFGY"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
Reference in a new issue View git blame Copy permalink