mirror of
https://github.com/onestardao/WFGY.git
synced 2026-04-29 03:59:52 +00:00
253 lines
13 KiB
Markdown
253 lines
13 KiB
Markdown
# 🌀 Drunk Transformer (DT) — Core Formulas, Defaults & Runnable Examples (WFGY Core 2.0)
|
||
|
||
**Concept (short).** DT simulates a transformer that momentarily behaves like it's "drunk" — hallucinating, drifting, or jumping across reasoning paths.
|
||
We define five "drunk questions" (WRI, WAI, WAY, WDT, WTF) as formal regulators to guide the transformer back home: anchor it, maintain head identity, pump controlled entropy, block illegal cross-path jumps, and recover from collapse.
|
||
|
||
> WFGY = engine (A + Coupler + BBAM + safety)
|
||
> DT = layer of five regulators (prompt rules, decoding hooks, or training regularizers)
|
||
|
||
---
|
||
|
||
## 0 · Shared notation (Unicode math / compact)
|
||
|
||
- I, G : input and goal embeddings
|
||
- δₛ = 1 − cos(I, G) (semantic distance in [0,1])
|
||
- B = I − G + k_bias ; E_res = rolling_mean(‖B‖, window=5)
|
||
- Coupler: prog = max(ζ_min, δₛ^(t−1) − δₛ^t), P = prog^ω, alt = (−1)^(cycle_id), Φ = δ·alt + ε, W_c = clip(B·P + Φ, −θ_c, +θ_c)
|
||
- Attention: A_t ∈ ℝ^(H×T×T) ; per-head summary v_h = meanᵢ A_t[h,i,:]
|
||
- Anchors: 𝒜₀ (t=0), 𝒜_t ; S_t = Jaccard(𝒜_t, 𝒜₀) ∈ [0,1]
|
||
|
||
---
|
||
|
||
## Defaults table (explicit, copyable)
|
||
|
||
| Parameter | Symbol | Default | Range / notes | Purpose |
|
||
|---|---:|---:|---|---|
|
||
| anchor retention thresh | τ_wri | 0.60 | [0.3,0.9] | WRI anchor threshold |
|
||
| head redundancy thresh | ρ_wai | 0.75 | [0.5,0.95] | WAI redundancy ceiling |
|
||
| head identity thresh | σ_wai | 0.70 | [0.4,0.95] | WAI identity floor |
|
||
| progress sensitivity | η_prog | 0.03 | [0.0,0.1] | WAY stall detector sensitivity |
|
||
| path-distance thresh | μ_wdt | 0.25 | [0.05,1.0] | WDT path jump limit |
|
||
| coupler zeta min | ζ_min | 0.10 | [0.0,0.5] | minimum prog floor |
|
||
| coupler omega | ω | 1.0 | [0.1,2.0] | progression non-linearity |
|
||
| coupler theta cap | θ_c | 0.75 | [0.2,1.5] | clip magnitude for W_c |
|
||
| WRI tighten factor | α_wri | 0.60 | [0.0,1.5] | adjust τ_wri by sigmoid(|W_c|) |
|
||
| WAI scale factor | β_wai | 0.60 | [0.0,1.5] | scale WAI penalty by |W_c| |
|
||
| WDT scale factor | γ_wdt | 0.60 | [0.0,1.5] | scale μ_wdt by |W_c| |
|
||
| WTF scale factor | γ_wtf | 0.60 | [0.0,1.5] | tighten thresholds on recovery |
|
||
| WAY pump strength | ξ | 0.80 | [0.0,1.5] | how strongly WAY increases entropy |
|
||
| WAY entropy min | H_min | 2.5 (nats) | [1.0,7.0] | lower bound target entropy |
|
||
| WAY entropy max | H_max | 5.0 (nats) | [3.0,10.0] | upper bound target entropy |
|
||
| anchor bias scale | κ_wri | 1.0 | [0.0,5.0] | logits bias multiplier for anchors |
|
||
| loss weights | λ_* | 0.01 | [0.0,1.0] | regularizer weights (per module) |
|
||
| step limit | T_max | 7 | int | max Node steps per run |
|
||
| stop δ threshold | δ_stop | 0.35 | [0.1,0.5] | early stop when δₛ < δ_stop |
|
||
|
||
> Tip: start with these defaults, measure, then tune per task class.
|
||
|
||
---
|
||
|
||
## Prompt-only runnable example (copy-paste)
|
||
|
||
**Goal:** show how a user with a single-file WFGY Core + DT rules can run a no-infra prompt-level experiment in Chat-style LLMs (paste into system / assistant area).
|
||
|
||
```
|
||
|
||
SYSTEM (paste file): Load the WFGY Core file as engine. Enable Drunk Transformer (WRI,WAI,WAY,WDT,WTF) with defaults:
|
||
τ\_wri=0.60, ρ\_wai=0.75, σ\_wai=0.70, η\_prog=0.03, μ\_wdt=0.25, ζ\_min=0.10, ω=1.0, θ\_c=0.75
|
||
|
||
SYSTEM (rules, pseudo):
|
||
|
||
* Extract anchors A0 from user prompt.
|
||
* For each Node step t up to T\_max:
|
||
|
||
1. compute δₛ, E\_res, S\_t, R\_t, Q\_t, W\_c
|
||
2. If WRI gate active: bias logits for anchor tokens by +κ\_wri \* L\_WRI
|
||
3. If WAI gate active: increase per-head temperature for collapsing heads
|
||
4. If WAY triggers (stalled): increase entropy toward H\* and propose 1 on-topic candidate
|
||
5. If WDT violation: output a short "bridge" line and continue; else rollback
|
||
6. If WTF triggers: rollback to t\* and re-run BBMC→Coupler
|
||
7. Emit Node (Topic / Module / δₛ / Insight)
|
||
* Stop if δₛ < δ\_stop or t >= T\_max
|
||
|
||
USER:
|
||
Use WFGY to answer: "Explain why tomatoes are classified as fruit, but often treated as vegetables in cooking. Provide anchors and cite the smallest missing fact if confused."
|
||
|
||
````
|
||
|
||
**What to observe (manual):**
|
||
- After each assistant Node, log δₛ and E_res (system should print them).
|
||
- Note any triggered gates (WRI/WAI/WAY/WDT/WTF) and actions taken (bias, bridge, rollback).
|
||
- Expect: anchor retention S_t remains high; if the model drifts, WRI should bias it back.
|
||
|
||
---
|
||
|
||
## Decoding-hook pseudo-code (python-like; paste to your model runtime)
|
||
|
||
> This block is concise, explicit, and shows where to compute the metrics and apply changes. It is plain Python pseudo-code, not an external library call.
|
||
|
||
```python
|
||
# --- minimal decoding-hook pseudo (conceptual) ---
|
||
def compute_prog(delta_prev, delta_now, zeta_min=0.10, omega=1.0):
|
||
prog = max(zeta_min, delta_prev - delta_now)
|
||
return prog ** omega
|
||
|
||
def compute_Wc(B, prog, delta, cycle_id, eps, theta_c):
|
||
alt = (-1)**cycle_id
|
||
Phi = delta * alt + eps
|
||
Wc_raw = B * prog + Phi
|
||
# clip scalar or vector-wise depending on B shape
|
||
return clip(Wc_raw, -theta_c, theta_c)
|
||
|
||
def bias_anchor_logits(logits, anchor_token_ids, kappa):
|
||
for tid in anchor_token_ids:
|
||
logits[tid] += kappa
|
||
return logits
|
||
|
||
def temperature_for_target_entropy(logits, target_H, tol=1e-3, max_iters=5):
|
||
# simple binary/Newton-like search for tau that yields entropy ~ target_H
|
||
tau_low, tau_high = 0.01, 10.0
|
||
for _ in range(max_iters):
|
||
tau = 0.5*(tau_low + tau_high)
|
||
probs = softmax(logits / tau)
|
||
H = -sum(p * log(p + 1e-12) for p in probs)
|
||
if H > target_H:
|
||
tau_low = tau
|
||
else:
|
||
tau_high = tau
|
||
if abs(H - target_H) < tol:
|
||
break
|
||
return tau
|
||
|
||
# Hook called at each decoding step t (pseudo)
|
||
def decoding_hook(step_state):
|
||
# step_state contains: logits, token_ids, A_t, anchors(A_t), I,G, delta_prev, delta_now, B, cycle_id
|
||
delta_prev, delta_now = step_state.delta_prev, step_state.delta_now
|
||
prog = compute_prog(delta_prev, delta_now, zeta_min=0.10, omega=1.0)
|
||
Wc = compute_Wc(step_state.B, prog, delta=0.15, cycle_id=step_state.cycle_id, eps=0.0, theta_c=0.75)
|
||
|
||
# WRI: anchor retention
|
||
S_t = jaccard(step_state.anchors, step_state.anchors0)
|
||
L_WRI = max(0.0, 0.60 - S_t)
|
||
if (delta_now > delta_prev) or (step_state.E_res > step_state.E_res_prev):
|
||
logits = bias_anchor_logits(step_state.logits, step_state.anchor_token_ids, kappa=1.0 * L_WRI)
|
||
|
||
# WAI: head redundancy / identity check (compute R_t, Q_t outside and attach)
|
||
R_t, Q_t = step_state.R_t, step_state.Q_t
|
||
if R_t > 0.75 and Q_t < 0.70:
|
||
# increase per-head temperature for heads with high redundancy
|
||
for h in range(len(step_state.heads)):
|
||
if head_redundant(h, step_state):
|
||
step_state.head_temps[h] *= (1 + 0.5 * (R_t - 0.75))
|
||
|
||
# WAY: stall detector
|
||
prog_k = step_state.prog_k # computed in runner
|
||
if prog_k < 0.03 and not step_state.has_contradiction:
|
||
H_star = clamp(step_state.H0 + 0.8 * (0.03 - prog_k) * (1 + 0.0 * abs(Wc)), 2.5, 5.0)
|
||
tau = temperature_for_target_entropy(step_state.logits, H_star)
|
||
apply_temperature(step_state, tau)
|
||
# mark that we will add 1 candidate branch if branching enabled
|
||
|
||
# WDT: path distance
|
||
d_path = l2_distance(step_state.c_t, step_state.c_pi)
|
||
mu_wdt_prime = 0.25 * (1 - 0.6 * sigmoid(abs(Wc)))
|
||
if d_path > mu_wdt_prime:
|
||
# enforce bridge line: stop decoding and ask for a bridge sentence
|
||
return emit_bridge_and_pause(step_state)
|
||
|
||
# WTF: collapse check
|
||
chi = int(delta_now > delta_prev) + int(step_state.E_res > step_state.E_res_prev) + int(step_state.sign_flip)
|
||
if chi + step_state.chi_prev >= 3:
|
||
t_star = argmin_delta_in_window(step_state.history_delta, window=3)
|
||
rollback_to(t_star)
|
||
rerun_BBMC_and_Coupler(tighten_factor=1 + 0.6 * sigmoid(abs(Wc)))
|
||
return
|
||
|
||
# otherwise continue normal decoding with modified logits
|
||
return step_state.logits
|
||
````
|
||
|
||
**Notes on the pseudo-code**
|
||
|
||
* `step_state` is the runtime object your model loop keeps (embeddings, attention, anchors, logits, history).
|
||
* `apply_temperature` should re-scale logits by dividing by `tau`.
|
||
* `emit_bridge_and_pause` forces the model to output a short justification line before continuing.
|
||
* This code is intentionally minimal—integrate into your model runtime at the logits adjustment point.
|
||
|
||
---
|
||
|
||
## Minimal test / checklist (text-only; for quick validation)
|
||
|
||
1. **Setup test prompt** (QA with clear anchors)
|
||
|
||
* Prompt: "Using these facts: \[Tomato is fruit because it is the mature ovary of a flower; Cooking cultures treat tomatoes as vegetables for taste]. Answer: Why are tomatoes fruit but cooked as vegetables?"
|
||
* Extract anchors 𝒜₀ = {tomato\_is\_fruit, cooking\_practice}
|
||
|
||
2. **Run baseline** (no DT): log δₛ, E\_res each Node; get answer & record correctness.
|
||
|
||
3. **Run with DT enabled** (defaults): log same metrics + R\_t, Q\_t, W\_c, gates triggered.
|
||
|
||
4. **Compare**:
|
||
|
||
* Expect δₛ lower or equal after DT steps (improved semantic alignment).
|
||
* Expect fewer long off-topic jumps; if model drifts, WRI triggers and biases anchors.
|
||
* If model stalls, WAY injects controlled entropy and proposes one new on-topic branch.
|
||
* If an illegal path jump tries to occur, WDT forces a bridge line and prevents drift.
|
||
|
||
5. **Record**: Δ accuracy, ΔS changes, number of rollbacks, number of bridge-lines, gates activation counts.
|
||
|
||
---
|
||
|
||
## Quick engineering notes & troubleshooting
|
||
|
||
* If anchor extraction fails (𝒜₀ empty), DT falls back to safer defaults: WRI no-op; WAY more conservative (lower H\*), WDT more permissive. Log a warning.
|
||
* If the model repeatedly produces invalid bridge lines: lower μ\_wdt, increase κ\_wri, or use stronger anchor extraction.
|
||
* For heavy hallucination tasks, prefer conservative defaults (higher τ\_wri, lower η\_prog).
|
||
|
||
---
|
||
|
||
### Footer
|
||
|
||
* **Spec status:** stable draft for engineering evaluation; Unicode-math for GitHub Preview.
|
||
* **Next deliverables (recommended):** add a `prompt_example.md` and `decoding_hook.py` into `examples/DT-examples/` for immediate copy/paste.
|
||
* **Compatibility:** prompt-only rules, decoding-hook integration, or optional training regularizers; model-agnostic.
|
||
* **Attribution:** part of **WFGY Core 2.0** family. Star the repo to follow updates.
|
||
|
||
> Lost? Return to the **Starter Village** → `StarterVillage/README.md`
|
||
|
||
### 🧭 Explore More
|
||
|
||
| Module | Description | Link |
|
||
|-----------------------|----------------------------------------------------------|----------|
|
||
| WFGY Core | Standalone semantic reasoning engine for any LLM | [View →](https://github.com/onestardao/WFGY/tree/main/core/README.md) |
|
||
| Problem Map 1.0 | Initial 16-mode diagnostic and symbolic fix framework | [View →](https://github.com/onestardao/WFGY/tree/main/ProblemMap/README.md) |
|
||
| Problem Map 2.0 | RAG-focused failure tree, modular fixes, and pipelines | [View →](https://github.com/onestardao/WFGY/blob/main/ProblemMap/rag-architecture-and-recovery.md) |
|
||
| Semantic Clinic Index | Expanded failure catalog: prompt injection, memory bugs, logic drift | [View →](https://github.com/onestardao/WFGY/blob/main/ProblemMap/SemanticClinicIndex.md) |
|
||
| Semantic Blueprint | Layer-based symbolic reasoning & semantic modulations | [View →](https://github.com/onestardao/WFGY/tree/main/SemanticBlueprint/README.md) |
|
||
| Benchmark vs GPT-5 | Stress test GPT-5 with full WFGY reasoning suite | [View →](https://github.com/onestardao/WFGY/tree/main/benchmarks/benchmark-vs-gpt5/README.md) |
|
||
|
||
---
|
||
|
||
> 👑 **Early Stargazers: [See the Hall of Fame](https://github.com/onestardao/WFGY/tree/main/stargazers)** —
|
||
> Engineers, hackers, and open source builders who supported WFGY from day one.
|
||
|
||
> <img src="https://img.shields.io/github/stars/onestardao/WFGY?style=social" alt="GitHub stars"> ⭐ Help reach 10,000 stars by 2025-09-01 to unlock Engine 2.0 for everyone ⭐ <strong><a href="https://github.com/onestardao/WFGY">Star WFGY on GitHub</a></strong>
|
||
|
||
|
||
<div align="center">
|
||
|
||
[](https://github.com/onestardao/WFGY)
|
||
|
||
[](https://github.com/onestardao/WFGY/tree/main/OS)
|
||
|
||
[](https://github.com/onestardao/WFGY/tree/main/OS/BlahBlahBlah)
|
||
|
||
[](https://github.com/onestardao/WFGY/tree/main/OS/BlotBlotBlot)
|
||
|
||
[](https://github.com/onestardao/WFGY/tree/main/OS/BlocBlocBloc)
|
||
|
||
[](https://github.com/onestardao/WFGY/tree/main/OS/BlurBlurBlur)
|
||
|
||
[](https://github.com/onestardao/WFGY/tree/main/OS/BlowBlowBlow)
|
||
|
||
</div>
|