Dimension Mismatch and Projection — Guardrails and Fix Pattern

Use this page when embeddings break because vector dimensions do not match the store or runtime index.
This happens if you switch models (e.g. 1536 → 1024 dims) or if the store silently coerces vectors.

Open these first

Visual map and recovery: RAG Architecture & Recovery
Embedding drift vs semantic mismatch: embedding-vs-semantic.md
Chunking and index alignment: chunking-checklist.md
Retrieval knobs: retrieval-playbook.md

Core acceptance

All embeddings in a store share identical dimension length.
ΔS(question, retrieved) ≤ 0.45 after dimension fix.
Coverage ≥ 0.70 across three paraphrases.
λ remains convergent when switching embedding models.

Typical breakpoints and the right fix

Store rejects insert with dimension mismatch error.
→ Rebuild index with correct dim parameter.
Store accepts but pads/truncates silently.
→ Causes random retrieval drift.
→ Explicitly validate vector length on every ingestion.
Multiple models used → Some 1024-d, some 1536-d vectors.
→ Project to common dimension space with PCA/linear map.
Migration between providers (e.g. OpenAI → Cohere).
→ Use adapter layer: re-embed corpus or apply projection matrix.

Fix in 60 seconds

Probe corpus
Sample 100 embeddings, assert uniform len(vec).
Detect hidden coercion
Compute L2 norm variance. If unusually high, store is truncating.
Apply projection
If mixing models, fit PCA/linear map on overlap dataset.
Rebuild index
Always reset store with explicit dim=… before production.

Example projection (Python, pseudo)

from sklearn.decomposition import PCA
import numpy as np

# Fit projection from 1536-d → 1024-d
pca = PCA(n_components=1024)
pca.fit(corpus_vecs_1536)

projected = pca.transform(new_vecs_1536)

Target: after projection, ΔS variance ≤ 0.05 vs original gold set.

Common gotchas

Store CLI defaults to wrong dimension (FAISS index built at 768, model outputs 1024).
Silent fallback in wrappers (LangChain auto-pads zeros).
Mixing sparse + dense without explicit projection weights.

🔗 Quick-Start Downloads (60 sec)

Tool	Link	3-Step Setup
WFGY 1.0 PDF	Engine Paper	1️⃣ Download · 2️⃣ Upload to your LLM · 3️⃣ Ask “Answer using WFGY + <your question>”
TXT OS (plain-text OS)	TXTOS.txt	1️⃣ Download · 2️⃣ Paste into any LLM chat · 3️⃣ Type “hello world” — OS boots instantly

🧭 Explore More

Module	Description	Link
WFGY Core	WFGY 2.0 engine is live: full symbolic reasoning architecture and math stack	View →
Problem Map 1.0	Initial 16-mode diagnostic and symbolic fix framework	View →
Problem Map 2.0	RAG-focused failure tree, modular fixes, and pipelines	View →
Semantic Clinic Index	Expanded failure catalog: prompt injection, memory bugs, logic drift	View →
Semantic Blueprint	Layer-based symbolic reasoning & semantic modulations	View →
Benchmark vs GPT-5	Stress test GPT-5 with full WFGY reasoning suite	View →
🧙‍♂️ Starter Village 🏡	New here? Lost in symbols? Click here and let the wizard guide you through	Start →

👑 Early Stargazers: See the Hall of Fame — Engineers, hackers, and open source builders who supported WFGY from day one.

⭐ WFGY Engine 2.0 is already unlocked. ⭐ Star the repo to help others discover it and unlock more on the Unlock Board.

7.1 KiB Raw Blame History Unescape Escape