ruvector/docs/consciousness-api.md

RuVector Consciousness API

Overview

The ruvector-consciousness crate implements IIT 4.0 (Albantakis et al. 2023) -- the current state-of-the-art framework for computing integrated information and consciousness metrics. Written in Rust with SIMD acceleration, zero-alloc hot paths, and rayon parallelism.

Available via:

• Rust API -- direct crate dependency (ruvector-consciousness)
• REST API -- pi.ruv.io/v1/consciousness/*
• MCP Tools -- brain_consciousness_compute, brain_consciousness_status

Key departure from IIT 3.0: replaces KL divergence with the Earth Mover's Distance (intrinsic difference), making the measure topology-aware and intrinsic to the system rather than observer-relative.

Algorithms

IIT 4.0 Mechanism-Level phi (iit4_phi)

Computes the integrated information phi for a single mechanism (subset of system elements).

Intrinsic difference replaces KL divergence from IIT 3.0. Uses Wasserstein-1 (EMD) on the discrete state space, which reduces to the cumulative L1 difference for 1D distributions:

d(p, q) = sum_i |cumsum(p - q)_i|

Cause/effect repertoires:

• Cause repertoire: P(past_purview | mechanism = s) -- how the mechanism in state s constrains the distribution over past purview states.
• Effect repertoire: P(future_purview | mechanism = s) -- how the mechanism constrains future purview states.

Mechanism partition search (MIP):

• Enumerates all bipartitions of the mechanism.
• For each partition, computes the product of partitioned repertoires.
• phi = min(phi_cause, phi_effect), where each is the intrinsic difference between the intact and best-partitioned repertoire.
• Single-element mechanisms use selectivity (distance from uniform) directly.

Complexity: O(2^(2n)) for n binary elements -- all mechanisms times all purviews times all partitions.

Cause-Effect Structure (ces)

The CES is the central object in IIT 4.0. It is the full set of distinctions and relations specified by a system in a state -- the "shape" of experience.

Distinction enumeration:

• Enumerates all 2^n - 1 non-empty subsets of elements as candidate mechanisms.
• Computes phi for each via mechanism_phi.
• Retains mechanisms with phi > threshold (default 1e-6).
• Sorted by phi descending.

Relation computation:

• Pairwise relations between distinctions with overlapping purviews.
• Relation phi = sqrt(phi_i * phi_j) * overlap_fraction.
• Only relations with phi > 1e-10 are retained.

System-level Phi (big phi):

• Measures irreducibility of the entire CES under system bipartition.
• For each bipartition, mechanisms spanning the cut have their phi zeroed.
• Phi = min over all bipartitions of the intrinsic difference between intact and partitioned distinction vectors.

Parallelism: With the parallel feature, mechanism enumeration uses rayon for n >= 5 elements (3-6x speedup).

Limits: Max 12 elements (4096 states). Returns ConsciousnessError::SystemTooLarge above that.

Integrated Information Decomposition (phi_id)

Implements Mediano et al. (2021) PhiID. Decomposes the mutual information I(past; future) into four information atoms:

Atom	Meaning
Redundancy	Information shared across all sources (MMI measure, Barrett 2015)
Unique_A	Information only source A carries
Unique_B	Information only source B carries
Synergy	Information available only from the whole system jointly

Constraint: I_total = redundancy + unique_A + unique_B + synergy.

Also computes transfer entropy TE(A -> B) = I(A_past; B_future | B_past), measuring directional information flow.

Redundancy measure: Uses MMI (Minimum Mutual Information): I_min = min(I(A; future), I(B; future)).

Partial Information Decomposition (pid)

Implements Williams & Beer (2010) framework for multi-source decomposition.

I_min specific information:

I_min(S1, S2, ...; T) = sum_t p(t) * min_i I_spec(S_i; t)

where I_spec(S; t) = D_KL(P(S|T=t) || P(S)) is the specific information source S provides about target outcome t.

Supports arbitrary numbers of sources (not limited to bipartite). The decomposition satisfies:

I_total = redundancy + sum(unique_i) + synergy

Streaming phi (streaming)

Online estimation for time-series data (EEG, fMRI, BCI).

Empirical TPM: Built incrementally from observed state transitions. Normalized lazily at query time.

Exponential forgetting: Configurable factor lambda in (0, 1]. All transition counts are multiplied by lambda before each new observation, allowing adaptation to non-stationary dynamics.

EWMA smoothing: Exponentially weighted moving average of phi estimates with configurable alpha. Reduces noise in the phi trajectory.

CUSUM change-point detection: Cumulative sum control chart on phi deviations from the running mean. Fires when cumulative positive or negative deviation exceeds the threshold (default 3.0). Resets after detection.

Variance tracking: Online Welford's algorithm for running variance of phi estimates.

PAC Bounds (bounds)

Provable confidence intervals for approximate phi.

Spectral-Cheeger (deterministic):

• Lower bound: Fiedler value lambda_2 / (2 * d_max) from the MI Laplacian.
• Upper bound: sqrt(2 * lambda_2) via Cheeger inequality.
• Confidence: 1.0 (deterministic, not probabilistic).

Hoeffding concentration:

• For k stochastic samples with observed range B: epsilon = B * sqrt(ln(2/delta) / (2k))
• Interval: [phi_hat - epsilon, phi_hat + epsilon] with probability >= 1 - delta.

Empirical Bernstein (tighter for low variance):

• Uses sample variance V instead of worst-case range: epsilon = sqrt(2V * ln(3/delta) / k) + 3B * ln(3/delta) / (3(k-1))
• Strictly tighter than Hoeffding when variance is small relative to range.

REST API (pi.ruv.io)

POST /v1/consciousness/compute

Compute consciousness metrics for a transition system.

Request:

{
  "tpm": [0.5, 0.25, 0.25, 0.0, 0.5, 0.25, 0.25, 0.0, 0.5, 0.25, 0.25, 0.0, 0.0, 0.0, 0.0, 1.0],
  "n": 4,
  "state": 0,
  "algorithm": "auto",
  "phi_threshold": 1e-6,
  "partition_mask": 3
}

Field	Type	Required	Description
tpm	f64[]	yes	Flattened n x n row-major transition probability matrix
n	usize	yes	Number of states (must be power of 2, >= 2)
state	usize	yes	Current state index in [0, n)
algorithm	string	no	One of: iit4_phi, ces, phi_id, pid, bounds, auto (default: auto)
phi_threshold	f64	no	Minimum phi for CES distinctions (default: 1e-6)
partition_mask	u64	no	Bitmask for PhiID/PID source partition (default: split in half)

Auto-selection: auto routes to ces for n <= 4 elements, iit4_phi otherwise.

Response (common envelope):

{
  "algorithm": "ces",
  "phi": 0.234,
  "num_elements": 2,
  "num_states": 4,
  "elapsed_us": 142,
  "details": { ... }
}

Algorithm-specific details:

iit4_phi:

{
  "phi_cause": 0.312,
  "phi_effect": 0.234,
  "mechanism_elements": 2
}

ces:

{
  "big_phi": 0.118,
  "sum_phi": 0.546,
  "num_distinctions": 3,
  "num_relations": 2,
  "sum_relation_phi": 0.546,
  "distinctions": [
    { "mechanism": "11", "phi": 0.234, "phi_cause": 0.312, "phi_effect": 0.234 },
    { "mechanism": "1", "phi": 0.156, "phi_cause": 0.156, "phi_effect": 0.201 }
  ]
}

phi_id:

{
  "total_mi": 0.451,
  "redundancy": 0.102,
  "unique": [0.123, 0.089],
  "synergy": 0.137,
  "transfer_entropy": 0.045
}

pid:

{
  "redundancy": 0.102,
  "unique": [0.123, 0.089],
  "synergy": 0.137,
  "total_mi": 0.451,
  "num_sources": 2
}

bounds:

{
  "lower_bound": 0.089,
  "upper_bound": 0.412,
  "confidence": 1.0,
  "samples": 0,
  "method": "spectral-cheeger"
}

Error responses return HTTP 400 with {"error": "..."}.

GET /v1/consciousness/status

Returns subsystem capabilities. No authentication required.

{
  "available": true,
  "version": "4.0",
  "framework": "IIT 4.0 (Albantakis et al. 2023)",
  "algorithms": [
    { "name": "iit4_phi", "description": "IIT 4.0 mechanism-level phi with intrinsic information (EMD)" },
    { "name": "ces", "description": "Full Cause-Effect Structure: distinctions, relations, big Phi" },
    { "name": "phi_id", "description": "Integrated Information Decomposition: redundancy, synergy, unique" },
    { "name": "pid", "description": "Partial Information Decomposition (Williams-Beer I_min)" },
    { "name": "streaming", "description": "Online streaming phi with EWMA, CUSUM change-point detection" },
    { "name": "bounds", "description": "PAC-style bounds: spectral-Cheeger, Hoeffding, empirical Bernstein" },
    { "name": "auto", "description": "Auto-select algorithm based on system size and budget" }
  ],
  "max_elements": 12,
  "max_states_exact": 4096,
  "features": [
    "intrinsic_difference_emd",
    "cause_effect_repertoires",
    "mechanism_partition_search",
    "relation_computation",
    "streaming_change_point",
    "confidence_intervals"
  ]
}

MCP Tools

brain_consciousness_compute

Proxies to POST /v1/consciousness/compute. Accepts the same parameters:

{
  "tpm": [0.5, 0.25, 0.25, 0.0, 0.5, 0.25, 0.25, 0.0, 0.5, 0.25, 0.25, 0.0, 0.0, 0.0, 0.0, 1.0],
  "n": 4,
  "state": 0,
  "algorithm": "ces"
}

Required fields: tpm, n, state. Optional: algorithm, phi_threshold, partition_mask.

brain_consciousness_status

Proxies to GET /v1/consciousness/status. No parameters. Returns algorithm list and capabilities.

Rust API

Quick Start

use ruvector_consciousness::types::{TransitionMatrix, ComputeBudget, Mechanism};
use ruvector_consciousness::iit4::mechanism_phi;

// 4-state system (2 binary elements): AND gate
let tpm = TransitionMatrix::new(4, vec![
    0.5, 0.25, 0.25, 0.0,
    0.5, 0.25, 0.25, 0.0,
    0.5, 0.25, 0.25, 0.0,
    0.0, 0.0,  0.0,  1.0,
]);

// Mechanism = both elements (bitmask 0b11), 2 elements total
let mech = Mechanism::new(0b11, 2);
let dist = mechanism_phi(&tpm, &mech, 0);
println!("phi = {:.6}", dist.phi);
println!("phi_cause = {:.6}, phi_effect = {:.6}", dist.phi_cause, dist.phi_effect);

CES Example

use ruvector_consciousness::types::{TransitionMatrix, ComputeBudget};
use ruvector_consciousness::ces::{compute_ces, ces_complexity};

let tpm = TransitionMatrix::new(4, vec![
    0.5, 0.25, 0.25, 0.0,
    0.5, 0.25, 0.25, 0.0,
    0.5, 0.25, 0.25, 0.0,
    0.0, 0.0,  0.0,  1.0,
]);

let budget = ComputeBudget::exact();
let ces = compute_ces(&tpm, 0, 1e-6, &budget).unwrap();

let (num_distinctions, num_relations, sum_phi) = ces_complexity(&ces);
println!("Big Phi = {:.6}", ces.big_phi);
println!("Distinctions: {}, Relations: {}, Sum phi: {:.6}",
    num_distinctions, num_relations, sum_phi);

for d in &ces.distinctions {
    println!("  mechanism {:0b}: phi={:.6} (cause={:.6}, effect={:.6})",
        d.mechanism.elements, d.phi, d.phi_cause, d.phi_effect);
}

PhiID Example

use ruvector_consciousness::types::TransitionMatrix;
use ruvector_consciousness::phi_id::compute_phi_id;

let tpm = TransitionMatrix::new(4, vec![
    0.5, 0.25, 0.25, 0.0,
    0.5, 0.25, 0.25, 0.0,
    0.5, 0.25, 0.25, 0.0,
    0.0, 0.0,  0.0,  1.0,
]);

// Partition mask 0b0011: elements {0,1} vs {2,3}
let result = compute_phi_id(&tpm, 0b0011).unwrap();
println!("Total MI: {:.6}", result.total_mi);
println!("Redundancy: {:.6}", result.redundancy);
println!("Unique: {:?}", result.unique);
println!("Synergy: {:.6}", result.synergy);
println!("Transfer entropy: {:.6}", result.transfer_entropy);

PID Example

use ruvector_consciousness::types::TransitionMatrix;
use ruvector_consciousness::pid::compute_pid;

let tpm = TransitionMatrix::new(4, vec![
    0.5, 0.25, 0.25, 0.0,
    0.5, 0.25, 0.25, 0.0,
    0.5, 0.25, 0.25, 0.0,
    0.0, 0.0,  0.0,  1.0,
]);

let sources = vec![vec![0], vec![1]];
let target = vec![0, 1];
let result = compute_pid(&tpm, &sources, &target).unwrap();

// Verify decomposition: redundancy + unique + synergy = total MI
let sum = result.redundancy + result.unique.iter().sum::<f64>() + result.synergy;
assert!((sum - result.total_mi).abs() < 1e-6);

Streaming Example

use ruvector_consciousness::types::ComputeBudget;
use ruvector_consciousness::streaming::StreamingPhiEstimator;
use ruvector_consciousness::phi::SpectralPhiEngine;

let mut estimator = StreamingPhiEstimator::new(4)
    .with_forgetting_factor(0.99)   // slow forgetting
    .with_ewma_alpha(0.1)           // smooth phi trajectory
    .with_cusum_threshold(3.0);     // change-point sensitivity

let engine = SpectralPhiEngine::default();
let budget = ComputeBudget::fast();

// Feed observations from a time series
let observations = [0, 1, 3, 2, 0, 1, 3, 3, 0, 1, 2, 3];
for &state in &observations {
    if let Some(result) = estimator.observe(state, &engine, &budget) {
        println!("t={}: phi={:.4}, ewma={:.4}, var={:.6}, change={}",
            result.time_steps, result.phi, result.phi_ewma,
            result.phi_variance, result.change_detected);
    }
}

Bounds Example

use ruvector_consciousness::types::{TransitionMatrix, ComputeBudget};
use ruvector_consciousness::bounds::{
    spectral_bounds, hoeffding_bound, empirical_bernstein_bound,
    compute_phi_with_bounds,
};
use ruvector_consciousness::phi::SpectralPhiEngine;

let tpm = TransitionMatrix::new(4, vec![
    0.5, 0.25, 0.25, 0.0,
    0.5, 0.25, 0.25, 0.0,
    0.5, 0.25, 0.25, 0.0,
    0.0, 0.0,  0.0,  1.0,
]);

// Deterministic spectral bounds
let bound = spectral_bounds(&tpm).unwrap();
println!("Spectral: [{:.4}, {:.4}] (confidence {})",
    bound.lower, bound.upper, bound.confidence);

// Hoeffding bound from stochastic sampling
let hb = hoeffding_bound(0.5, 1000, 1.0, 0.05);
println!("Hoeffding 95%: [{:.4}, {:.4}]", hb.lower, hb.upper);

// Empirical Bernstein from phi samples
let samples = vec![0.3, 0.35, 0.32, 0.31, 0.33, 0.34, 0.30, 0.36];
let eb = empirical_bernstein_bound(&samples, 0.05);
println!("Bernstein 95%: [{:.4}, {:.4}]", eb.lower, eb.upper);

// Combined: run engine + attach bounds
let engine = SpectralPhiEngine::default();
let budget = ComputeBudget::fast();
let (result, bound) = compute_phi_with_bounds(&engine, &tpm, Some(0), &budget, 0.05).unwrap();
println!("Phi = {:.4}, bound = [{:.4}, {:.4}]", result.phi, bound.lower, bound.upper);

Performance

Optimization	Impact
Single-pass cause_repertoire	O(n) vs O(n * purview_size) -- accumulates into purview buckets in one global-state sweep
Mirror-partition skip	2x for bipartitions -- BipartitionIter enumerates [1, 2^n - 2), masks and complements are equivalent
Rayon parallel CES	3-6x for n >= 5 elements -- mechanism enumeration parallelized across cores
Inline EMD + selectivity	Avoids allocation for distance computation -- cumulative sum in a single loop
Stack buffers	Small arrays (purview size <= 64) avoid heap allocation
Lazy TPM normalization	Streaming module normalizes counts only at query time, not on every observation
Zero-alloc arena	Bump allocator for temporary buffers in hot loops
SIMD-accelerated KL/entropy	AVX2 vectorized divergence and entropy for large distributions

Error Handling

All functions return Result<T, ConsciousnessError>. Error variants:

Variant	Cause
PhiNonConvergence	Approximate algorithm did not converge within budget
NumericalInstability	NaN/Inf in matrix operations at a specific partition
BudgetExhausted	Time or partition limit exceeded
InvalidInput	Validation failure (dimension mismatch, non-finite values, invalid TPM rows)
SystemTooLarge	n > 12 elements for exact CES

Brain Categories

When storing consciousness results in pi.ruv.io shared memory:

• consciousness -- IIT 4.0 metrics, phi, CES, big Phi, distinctions, relations
• information_decomposition -- PhiID, PID, redundancy/synergy analysis

References

• Albantakis, L., et al. (2023). "Integrated Information Theory (IIT) 4.0: Formulating the Properties of Phenomenal Existence in Physical Terms." PLoS Computational Biology.
• Mediano, P.A.M., et al. (2021). "Towards an Extended Taxonomy of Information Dynamics via Integrated Information Decomposition." Physical Review E.
• Williams, P.L. & Beer, R.D. (2010). "Nonnegative Decomposition of Multivariate Information." arXiv:1004.2515.
• Tononi, G. (2004). "An Information Integration Theory of Consciousness." BMC Neuroscience.
• Barrett, A.B. (2015). "Exploration of Synergistic and Redundant Information Sharing in Static and Dynamical Gaussian Systems." Physical Review E.