vrr/WFGY
2
0
Fork 0
mirror of https://github.com/onestardao/WFGY.git synced 2026-07-09 15:58:34 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 4
WFGY/Polaris/experiments/downloads/PP02D.ipynb
PSBigBig + MiniPS c5652e7302
Add files via upload
2026-05-06 17:23:22 +08:00

450 lines
17 KiB
Text
Raw Permalink Blame History

{
"cells": [
{
"cell_type": "markdown",
"id": "3db2a65b",
"metadata": {},
"source": [
"# PP02D_A — API Stability 216 Fixtures\n",
"\n",
"**Belongs to:** WFGY 5.0 Polaris Protocol \n",
"**Repository:** https://github.com/onestardao/WFGY \n",
"**Polaris Protocol path:** https://github.com/onestardao/WFGY/tree/main/Polaris \n",
"**Experiment page:** https://github.com/onestardao/WFGY/blob/main/Polaris/experiments/README.md\n",
"\n",
"## Experiment spirit\n",
"\n",
"API stability supplement. This checks structured-output preservation, not real-world API guarantees or global QA improvement.\n",
"\n",
"## What this Colab tests\n",
"\n",
"Reviews structured-output preservation and support closure across a 216-fixture, 6-shard API-style validation path.\n",
"\n",
"## Published result summary\n",
"\n",
"- 216 fixtures; 6 shards.\n",
"- 0 API exceptions; 0 parse failures.\n",
"- 216/216 fixture pass.\n",
"- 1944/1944 semantic support checks; 2160/2160 support closure checks; 12/12 poison reject.\n",
"\n",
"## Expected usage / token-cost note\n",
"\n",
"Review Colab can run without model calls when inspecting packaged results. Full adaptation may require API usage depending on selected settings.\n",
"\n",
"## Scientific boundary\n",
"\n",
"Structured API-style stability supplement only. Not proof that all API usage is stable.\n",
"\n",
"This Colab is provided for **reproduction, inspection, and adaptation**. You do not have to rerun it to read the published result summary. If you rerun it, model behavior, token usage, cost, and output details may vary.\n",
"\n",
"---\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "1e3f7c74",
"metadata": {},
"outputs": [],
"source": [
"# Polaris Protocol experiment identity\n",
"POLARIS_PROTOCOL_REPO = \"https://github.com/onestardao/WFGY\"\n",
"POLARIS_EXPERIMENT_ID = \"PP02D_A\"\n",
"POLARIS_EXPERIMENT_TITLE = \"PP02D_A — API Stability 216 Fixtures\"\n",
"POLARIS_EXPERIMENT_ROLE = \"API Stability 216 Fixtures\"\n",
"POLARIS_CLAIM_BOUNDARY = \"Structured API-style stability supplement only. Not proof that all API usage is stable.\"\n",
"print(f\"{POLARIS_EXPERIMENT_ID} — {POLARIS_EXPERIMENT_TITLE}\")\n",
"print(f\"Repo: {POLARIS_PROTOCOL_REPO}\")\n"
]
},
{
"cell_type": "markdown",
"id": "bd2b9783",
"metadata": {},
"source": [
"# WFGY 5.0 Polaris Protocol — DD02A Review Notebook\n",
"\n",
"Repository: https://github.com/onestardao/WFGY\n",
"\n",
"This notebook reviews the completed DD02A structural API validation package. It summarizes the 216-fixture sharded run, parser checks, poison rejection checks, and package integrity audit.\n",
"\n",
"Claim boundary: this notebook validates the structural pipeline evidence package. It does not claim real QA accuracy improvement. The recommended next branch is DD02C real QA validation.\n"
]
},
{
"cell_type": "markdown",
"id": "8e6e1a2a",
"metadata": {},
"source": [
"## How to use\n",
"\n",
"Place the source artifact zip in the same directory as this notebook, or upload it when prompted. The expected filename is:\n",
"\n",
"```text\n",
"PP02D_A_API_FULL_216_SHARDED_GREEN_PP02D_A_API_MVP_20260504T150642Z.zip\n",
"```\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "c27d54f8",
"metadata": {},
"outputs": [],
"source": [
"import os, json, zipfile, hashlib, csv, textwrap\n",
"from pathlib import Path\n",
"import pandas as pd\n",
"\n",
"ZIP_NAME = 'PP02D_A_API_FULL_216_SHARDED_GREEN_PP02D_A_API_MVP_20260504T150642Z.zip'\n",
"zip_path = Path(ZIP_NAME)\n",
"if not zip_path.exists():\n",
" try:\n",
" from google.colab import files\n",
" uploaded = files.upload()\n",
" if ZIP_NAME not in uploaded:\n",
" raise FileNotFoundError(f'Expected {ZIP_NAME}, but uploaded: {list(uploaded)}')\n",
" zip_path = Path(ZIP_NAME)\n",
" except Exception as exc:\n",
" raise FileNotFoundError(f'Place {ZIP_NAME} next to this notebook or upload it in Colab.') from exc\n",
"\n",
"extract_dir = Path('dd02a_extracted_results')\n",
"extract_dir.mkdir(exist_ok=True)\n",
"with zipfile.ZipFile(zip_path, 'r') as zf:\n",
" zf.extractall(extract_dir)\n",
"print('Extracted files:', len(list(extract_dir.iterdir())))\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "c5baac25",
"metadata": {},
"outputs": [],
"source": [
"def load_json(name):\n",
" with open(extract_dir / name, 'r', encoding='utf-8') as f:\n",
" return json.load(f)\n",
"\n",
"score = load_json('api_score_summary.json')\n",
"integrity = load_json('api_package_integrity_audit.json')\n",
"preflight = load_json('preflight_gate_record.json')\n",
"run_config = load_json('run_config.json')\n",
"\n",
"summary_rows = [\n",
" ('Run mode', score.get('api_run_mode')),\n",
" ('Fixture count', score.get('fixture_count')),\n",
" ('Shard count', score.get('shard_count')),\n",
" ('API exceptions', score.get('api_exception_count')),\n",
" ('Shard parse failures', score.get('shard_parse_fail_count')),\n",
" ('Parse failures', score.get('parse_fail_count')),\n",
" ('Fixture pass', score.get('fixture_pass_count')),\n",
" ('Semantic citation checks', score.get('semantic_citation_checks')),\n",
" ('Aggregate support union checks', score.get('aggregate_support_union_checks')),\n",
" ('Total support closure', score.get('total_support_closure')),\n",
" ('Counterfactual pairs', score.get('counterfactual_pairs_pass')),\n",
" ('Poison rejection', preflight.get('poison_reject_count')),\n",
" ('Sandbox A', preflight.get('sandbox_pass_a', {}).get('status')),\n",
" ('Sandbox B', preflight.get('sandbox_pass_b', {}).get('status')),\n",
" ('Package self-check', integrity.get('zip_self_check_pass')),\n",
" ('Final verdict', score.get('api_verdict')),\n",
"]\n",
"summary_df = pd.DataFrame(summary_rows, columns=['Metric', 'Result'])\n",
"summary_df\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "2a49653b",
"metadata": {},
"outputs": [],
"source": [
"sandbox_df = pd.read_csv(extract_dir / 'full216_sharded_sandbox_ab_results.csv')\n",
"poison_df = pd.read_csv(extract_dir / 'full216_sharded_parser_poison_results.csv')\n",
"parse_df = pd.read_csv(extract_dir / 'shard_parse_trace.csv')\n",
"budget_df = pd.read_csv(extract_dir / 'shard_schema_budget_audit.csv')\n",
"display(sandbox_df)\n",
"display(poison_df)\n",
"display(parse_df)\n",
"display(budget_df)\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "6bc1d361",
"metadata": {},
"outputs": [],
"source": [
"checks = {\n",
" 'api_verdict_green': score.get('api_verdict') == 'GREEN',\n",
" 'fixtures_216': score.get('fixture_count') == 216,\n",
" 'six_shards': score.get('shard_count') == 6,\n",
" 'no_api_exceptions': score.get('api_exception_count') == 0,\n",
" 'no_parse_failures': score.get('parse_fail_count') == 0 and score.get('shard_parse_fail_count') == 0,\n",
" 'fixture_pass_216_216': score.get('fixture_pass_count') == '216/216',\n",
" 'poison_12_12': preflight.get('poison_reject_count') == '12/12',\n",
" 'package_self_check': integrity.get('zip_self_check_pass') is True,\n",
" 'no_hash_mismatches': integrity.get('hash_mismatches') == [],\n",
"}\n",
"pd.DataFrame(checks.items(), columns=['Check', 'Pass'])\n"
]
},
{
"cell_type": "markdown",
"id": "200c4699",
"metadata": {},
"source": [
"## Interpretation\n",
"\n",
"DD02A is a successful structural validation run. The package supports a green claim for strict structured-output pipeline stability.\n",
"\n",
"The correct next validation is DD02C, a 20 to 30 case real QA test that compares baseline raw-context answering against Polaris compact/topology answering.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "b8a86f67",
"metadata": {},
"outputs": [],
"source": [
"report = {\n",
" 'branch_alias': 'DD02A',\n",
" 'source_run_id': run_config.get('run_id'),\n",
" 'status': score.get('api_verdict'),\n",
" 'safe_claim': 'DD02A validates structured API pipeline stability and strict evidence-state preservation.',\n",
" 'not_claimed': 'DD02A does not claim real QA accuracy improvement.',\n",
" 'next_branch': 'DD02C_REAL_QA_30'\n",
"}\n",
"with open('DD02A_review_summary.json', 'w', encoding='utf-8') as f:\n",
" json.dump(report, f, ensure_ascii=False, indent=2)\n",
"print(json.dumps(report, ensure_ascii=False, indent=2))\n"
]
},
{
"cell_type": "markdown",
"id": "9b90af7c",
"metadata": {},
"source": [
"---\n",
"\n",
"# Final Polaris Protocol report\n",
"\n",
"This final section generates a compact report for **PP02D_A — API Stability 216 Fixtures**.\n",
"\n",
"It produces:\n",
"- `polaris_report_PP02D_A/executive_report.md`\n",
"- `polaris_report_PP02D_A/metrics_summary.csv`\n",
"- `polaris_report_PP02D_A/chart_*.png`\n",
"\n",
"The charts are intentionally simple. They are meant to make the experiment result easy to inspect, not to expand the claim boundary.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "0a0c3db5",
"metadata": {},
"outputs": [],
"source": [
"# Final Polaris Protocol report generator\n",
"# This cell uses the published result summary as a stable reporting fallback.\n",
"# If you adapt the notebook and compute fresh metrics, update POLARIS_PUBLISHED_METRICS before running this cell.\n",
"\n",
"from pathlib import Path\n",
"import json, csv, math, re\n",
"import pandas as pd\n",
"import matplotlib.pyplot as plt\n",
"\n",
"POLARIS_REPORT_EXPERIMENT_ID = \"PP02D_A\"\n",
"POLARIS_REPORT_TITLE = \"PP02D_A — API Stability 216 Fixtures\"\n",
"POLARIS_REPORT_REPO = \"https://github.com/onestardao/WFGY\"\n",
"POLARIS_REPORT_PURPOSE = \"Reviews structured-output preservation and support closure across a 216-fixture, 6-shard API-style validation path.\"\n",
"POLARIS_REPORT_SPIRIT = \"API stability supplement. This checks structured-output preservation, not real-world API guarantees or global QA improvement.\"\n",
"POLARIS_REPORT_CLAIM_BOUNDARY = \"Structured API-style stability supplement only. Not proof that all API usage is stable.\"\n",
"POLARIS_PUBLISHED_SUMMARY = [\n",
" \"216 fixtures; 6 shards.\",\n",
" \"0 API exceptions; 0 parse failures.\",\n",
" \"216/216 fixture pass.\",\n",
" \"1944/1944 semantic support checks; 2160/2160 support closure checks; 12/12 poison reject.\"\n",
"]\n",
"POLARIS_PUBLISHED_METRICS = {\n",
" \"Fixtures\": 216,\n",
" \"Shards\": 6,\n",
" \"API exceptions\": 0,\n",
" \"Parse failures\": 0,\n",
" \"Fixture pass\": 216,\n",
" \"Fixture total\": 216,\n",
" \"Semantic support checks\": 1944,\n",
" \"Semantic support total\": 1944,\n",
" \"Support closure checks\": 2160,\n",
" \"Support closure total\": 2160,\n",
" \"Poison reject\": 12,\n",
" \"Poison total\": 12\n",
"}\n",
"POLARIS_CHART_SPECS = [\n",
" {\n",
" \"title\": \"Fixture pass\",\n",
" \"ylabel\": \"Fixtures\",\n",
" \"labels\": [\n",
" \"Pass\",\n",
" \"Total\"\n",
" ],\n",
" \"values\": [\n",
" 216,\n",
" 216\n",
" ]\n",
" },\n",
" {\n",
" \"title\": \"Support checks\",\n",
" \"ylabel\": \"Checks\",\n",
" \"labels\": [\n",
" \"Semantic support\",\n",
" \"Support closure\"\n",
" ],\n",
" \"values\": [\n",
" 1944,\n",
" 2160\n",
" ]\n",
" },\n",
" {\n",
" \"title\": \"Poison rejection\",\n",
" \"ylabel\": \"Count\",\n",
" \"labels\": [\n",
" \"Rejected\",\n",
" \"Total\"\n",
" ],\n",
" \"values\": [\n",
" 12,\n",
" 12\n",
" ]\n",
" }\n",
"]\n",
"\n",
"report_dir = Path(f\"polaris_report_{POLARIS_REPORT_EXPERIMENT_ID}\")\n",
"report_dir.mkdir(exist_ok=True)\n",
"\n",
"# Save metrics table.\n",
"metrics_df = pd.DataFrame([\n",
" {\"metric\": key, \"value\": value}\n",
" for key, value in POLARIS_PUBLISHED_METRICS.items()\n",
"])\n",
"metrics_path = report_dir / \"metrics_summary.csv\"\n",
"metrics_df.to_csv(metrics_path, index=False)\n",
"\n",
"# Generate compact charts.\n",
"chart_paths = []\n",
"def _safe_chart_name(title):\n",
" return re.sub(r\"[^a-z0-9_\\\\-]+\", \"_\", title.lower().replace(\" \", \"_\")).strip(\"_\")\n",
"\n",
"for idx, spec in enumerate(POLARIS_CHART_SPECS, start=1):\n",
" labels = spec[\"labels\"]\n",
" values = spec[\"values\"]\n",
" plt.figure(figsize=(max(6, len(labels) * 1.2), 4))\n",
" bars = plt.bar(labels, values)\n",
" plt.title(spec[\"title\"])\n",
" plt.ylabel(spec.get(\"ylabel\", \"Value\"))\n",
" plt.xticks(rotation=25, ha=\"right\")\n",
"\n",
" numeric_values = [float(v) for v in values if isinstance(v, (int, float))]\n",
" max_value = max(numeric_values) if numeric_values else 0.0\n",
" min_value = min(numeric_values) if numeric_values else 0.0\n",
"\n",
" if max_value == 0 and min_value == 0:\n",
" # Make zero-count success charts visually readable instead of looking blank.\n",
" plt.ylim(0, 1)\n",
" for bar, value in zip(bars, values):\n",
" x = bar.get_x() + bar.get_width() / 2\n",
" plt.text(x, 0.05, str(value), ha=\"center\", va=\"bottom\", fontsize=10)\n",
" plt.text(\n",
" 0.5, 0.88,\n",
" \"All tracked counts are 0\",\n",
" transform=plt.gca().transAxes,\n",
" ha=\"center\",\n",
" va=\"center\",\n",
" fontsize=11,\n",
" )\n",
" else:\n",
" upper = max_value * 1.18 if max_value > 0 else 1\n",
" if min_value >= 0:\n",
" plt.ylim(0, upper)\n",
" for bar, value in zip(bars, values):\n",
" x = bar.get_x() + bar.get_width() / 2\n",
" y = bar.get_height()\n",
" label = f\"{value:.4g}\" if isinstance(value, float) else str(value)\n",
" plt.text(x, y + (upper * 0.02 if max_value > 0 else 0.03), label, ha=\"center\", va=\"bottom\", fontsize=9)\n",
"\n",
" plt.tight_layout()\n",
" chart_path = report_dir / f\"chart_{idx:02d}_{_safe_chart_name(spec['title'])}.png\"\n",
" plt.savefig(chart_path, dpi=180, bbox_inches=\"tight\")\n",
" plt.show()\n",
" chart_paths.append(chart_path)\n",
"\n",
"# Save executive markdown report.\n",
"summary_lines = \"\\n\".join([f\"- {item}\" for item in POLARIS_PUBLISHED_SUMMARY])\n",
"chart_lines = \"\\n\".join([f\"- {p.name}\" for p in chart_paths])\n",
"\n",
"report_md = f\"\"\"# {POLARIS_REPORT_TITLE}\n",
"\n",
"**Repository:** {POLARIS_REPORT_REPO} \n",
"**Experiment ID:** {POLARIS_REPORT_EXPERIMENT_ID}\n",
"\n",
"## Experiment spirit\n",
"\n",
"{POLARIS_REPORT_SPIRIT}\n",
"\n",
"## What this tests\n",
"\n",
"{POLARIS_REPORT_PURPOSE}\n",
"\n",
"## Published result summary\n",
"\n",
"{summary_lines}\n",
"\n",
"## Generated files\n",
"\n",
"- metrics_summary.csv\n",
"{chart_lines}\n",
"\n",
"## Claim boundary\n",
"\n",
"{POLARIS_REPORT_CLAIM_BOUNDARY}\n",
"\n",
"This report is designed for reproduction / inspection. It does not convert scoped evidence into universal proof.\n",
"\"\"\"\n",
"\n",
"report_path = report_dir / \"executive_report.md\"\n",
"report_path.write_text(report_md, encoding=\"utf-8\")\n",
"\n",
"print(\"Polaris Protocol report generated:\")\n",
"print(f\"- {report_path}\")\n",
"print(f\"- {metrics_path}\")\n",
"for path in chart_paths:\n",
" print(f\"- {path}\")\n"
]
}
],
"metadata": {
"colab": {
"provenance": []
},
"kernelspec": {
"display_name": "Python 3",
"name": "python3"
},
"language_info": {
"name": "python"
},
"polaris_protocol": {
"claim_boundary": "Structured API-style stability supplement only. Not proof that all API usage is stable.",
"experiment_id": "PP02D_A",
"experiment_title": "PP02D_A — API Stability 216 Fixtures",
"mvp_official_colab": true,
"repo": "https://github.com/onestardao/WFGY"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
Reference in a new issue View git blame Copy permalink