mirror of
https://github.com/ruvnet/RuView.git
synced 2026-05-20 09:32:59 +00:00
27a6edba8b
* feat(examples/three.js): cinematic skinned realtime pose demo + ESP32 CSI bridge
Five-stage example progression exploring three.js helpers (ADR-097 surface) as
a viewer for live RuView sensor data:
1. helpers-demo.html — clean ADR-097 helper reference (GridHelper,
PolarGridHelper, BoxHelper, AxesHelper),
file://-safe, no backend
2. helpers-cinematic.html — same scene + UnrealBloomPass + pseudo-CSI
sonar pings + tomography sweep + procedural
cyber floor + ambient drift particles
3. helpers-skinned.html — replaces sphere skeleton with Mixamo X Bot
via GLTFLoader from threejs.org CDN, plays
bundled animations with additive blending
4. helpers-skinned-fbx.html — same but loads a local Mixamo FBX (needs
serve-demo.py — file:// can't fetch local
siblings). Drop X Bot.fbx alongside.
5. helpers-skinned-realtime.html — webcam → MediaPipe Pose Heavy →
poseWorldLandmarks → direct quaternion
retargeting onto the Mixamo skeleton.
Real ESP32-S3 CSI streamed over WebSocket
from ruvultra (Tailscale, port 8766).
Supporting:
- serve-demo.py threaded HTTP server with no-cache headers
(fixes net::ERR_EMPTY_RESPONSE on the FBX path)
- ruvultra-csi-bridge.py ESP32 RuView firmware tick → WebSocket bridge,
runs as systemd-run unit on ruvultra
Bugs found + fixed along the way (all documented in code comments):
- FBX exports yield TWO parallel Bone trees with identical names; only the
SkinnedMesh.skeleton.bones one drives visible deformation. model.traverse
finds orphans.
- Mixamo FBX nests a zero-length wrapper bone above the real bone, same name.
bone.children[0].getWorldPosition == bone.getWorldPosition → restDir is
(0,0,0) → setFromUnitVectors collapses to identity. Walk past same-named
same-position wrappers when computing tail.
- AnimationMixer.update() with a "stopped" action still mutates bones unless
enabled=false is set.
Retargeting layer in helpers-skinned-realtime.html:
- 12 bones direct quaternion retarget (arms × 2, legs × 2, spine × 3, neck)
- Hips root rotation from shoulder/hip line basis (torso twist + lean)
- Neck aims at ear-midpoint (kp 7+8), not nose (kp 0), to remove the
forward bias of the protruding-nose anchor
- One Euro Filter per landmark per axis (Casiez 2012) — adaptive low-pass
- Visibility-weighted per-bone slerp gain — occluded limbs relax to rest
- URL toggles: ?mirror= ?yflip= ?zflip= ?cnn=0/1/2 ?csi=ws://...
Live CSI integration:
- Bridge parses adaptive_ctrl tick lines (motion/presence/rssi/yield)
- Browser fans single ESP32 reading across 4 UI nodes with phase-shifted
wobble (0.88–1.00 × sin(t·0.55 + offsetᵢ))
- EMA α=0.06 (~3 sec time constant), HUD update throttled 3 Hz
Co-Authored-By: claude-flow <ruv@ruv.net>
* refactor(examples/three.js): organize into demos/screenshots/server/assets + add README
Flatten the 13-file flat layout into purposeful subfolders so the demo
collection has a clean top-level entry point (README.md) and the file roles
are obvious from a directory listing.
Layout:
demos/ 01..05 — numbered for the progression (helpers → cinematic →
skinned → skinned-fbx → skinned-realtime)
screenshots/ one PNG per demo, matching the demo's filename prefix
server/ serve-demo.py + ruvultra-csi-bridge.py
assets/ X Bot.fbx (gitignored, used by demos 04 and 05)
Touched files (beyond the renames):
- 04-skinned-fbx.html, 05-skinned-realtime.html: MODEL_URL now resolves
'../assets/X%20Bot.fbx' instead of './X%20Bot.fbx'
- server/serve-demo.py: chdir() walks 3 levels up to repo root (was 2), and
the URL banner now lists all 5 demos
- .gitignore: comment refresh — points at assets/ and screenshots/
- 05-skinned-realtime.html also picks up in-flight fps-tune work from this
branch (Holistic script, SMOOTH_K URL param, slerp gain scaling) since
those edits and the rename hit the same file
Verified end-to-end:
- python examples/three.js/server/serve-demo.py
- all 5 demos return 200, X Bot.fbx returns 200 from new asset/ path
- demos 04 + 05 render the X Bot mesh; 0 JS errors via browser eval
- screenshots reproduced match the originals
Co-Authored-By: claude-flow <ruv@ruv.net>
|
||
|---|---|---|
| .. | ||
| demos | ||
| screenshots | ||
| server | ||
| .gitignore | ||
| README.md | ||
three.js demos
Five progressively richer browser demos of the ADR-097 sensing-helpers scene, ending with a live MediaPipe-Pose → Mixamo X Bot retargeting pipeline driven by a real ESP32 CSI feed.
Run them
python examples/three.js/server/serve-demo.py
# then open one of the URLs the script prints
server/serve-demo.py is a tiny ThreadingHTTPServer with aggressive
no-cache headers — the stdlib http.server is single-threaded and times out
on the parallel script + FBX fetches the demos make.
Demos
| # | File | What it shows |
|---|---|---|
| 01 | demos/01-helpers.html |
Plain ADR-097 helpers in the point-cloud viewer |
| 02 | demos/02-cinematic.html |
Cinematic camera + pseudo-CSI visualization on top of #01 |
| 03 | demos/03-skinned.html |
GLTF skinned mesh + additive animation blending |
| 04 | demos/04-skinned-fbx.html |
Mixamo X Bot loaded from FBX in the ADR-097 scene |
| 05 | demos/05-skinned-realtime.html |
Webcam → MediaPipe Pose Heavy → Mixamo IK retarget, live ESP32 CSI overlay |
| Screenshot | |
|---|---|
 |
 |
 |
 |
 |
Layout
examples/three.js/
├── README.md
├── .gitignore
├── demos/ # 5 self-contained HTML demos
│ ├── 01-helpers.html
│ ├── 02-cinematic.html
│ ├── 03-skinned.html
│ ├── 04-skinned-fbx.html
│ └── 05-skinned-realtime.html
├── screenshots/ # one PNG per demo
│ └── 0N-*.png
├── server/
│ ├── serve-demo.py # local HTTP server with no-cache headers
│ └── ruvultra-csi-bridge.py # ESP32 CSI WebSocket bridge (ruvultra:8766)
└── assets/
└── X Bot.fbx # gitignored — get your own from mixamo.com
# (FBX Binary, T-Pose, Without Skin)
# used by demos 04 and 05
Mixamo X Bot
Demos 04 and 05 expect assets/X Bot.fbx. It's gitignored (size + license
boundary). Download yours from mixamo.com: pick the
"X Bot" character, export as FBX Binary, T-Pose, Without Skin,
and drop it into assets/.
Live ESP32 CSI overlay (demo 05 only)
server/ruvultra-csi-bridge.py is the systemd-deployable bridge that runs on
the ruvultra host (over Tailscale). It listens for ESP32-S3 CSI on UDP and
re-broadcasts it as WebSocket frames at ws://ruvultra:8766/csi. Demo 05
auto-connects; if the socket is down, it falls back to the bundled idle clip
plus a synthetic CSI driver.
Open issues
- #583 — head/face tracking
fidelity in
05-skinned-realtime.html. Recommended fix: swap MediaPipe Pose Heavy for MediaPipe Holistic (same API, adds 468-point face mesh + hand landmarks for proper PnP head pose and finger curl tracking).