math-pixel/La-Fabrik
1
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity

feat add left hand tracking glove model

Browse Source
This commit is contained in:
Tom Boullay
2026-05-02 00:14:56 +02:00
parent 2aa662669f
commit 8fbb2e9428
8 changed files with 156 additions and 6 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/technical/hand-tracking.md
+11 -2
View File
@@ -16,6 +16,7 @@ The feature is scoped to the debug physics scene rather than production gameplay
4. The backend returns hand data including landmarks, handedness, score, center point, and `isFist`.
5. React stores the latest snapshot in the hand tracking provider.
6. `GrabbableObject` reads that snapshot each frame and uses fist state plus raycasting to grab objects.
7. `HandTrackingLeftGlove` reads the same snapshot and places the rigged `gant_l` model on the detected left hand in the debug physics scene.
## Activation Rules
@@ -104,12 +105,19 @@ The final hold distance is clamped between the configured grab minimum and maxim
The current debug UI includes:
- `HandTrackingDebugPanel` inside `DebugOverlayLayout` for status, usage, model-loaded placeholder, server state, hand count, and fist state
- `HandTrackingDebugPanel` inside `DebugOverlayLayout` for status, usage, loaded glove model, server state, hand count, and fist state
- `HandTrackingVisualizer` for the SVG landmark wireframe
- `HandTrackingLeftGlove` for the left-hand `gant_l` model in the R3F scene
- `r3f-perf` for render performance
- `lil-gui` for scene, camera, lighting, interaction, and grab controls
The hand tracking debug panel is a compact HTML grid outside the canvas. `Model loaded` is currently hardcoded to `none` until model-loading information is wired into the hand tracking flow. The hand wireframe is also HTML/SVG, not a 3D hand model.
The hand tracking debug panel is a compact HTML grid outside the canvas. `Model loaded` displays `gant_l` when a left hand is detected, otherwise `none`. The hand wireframe is also HTML/SVG, not a 3D hand model.
## Left Glove Model
The current left glove MVP uses `public/models/gant_l/model.gltf`, which contains a GLTF skin and armature. For now the model is positioned, oriented, and scaled as a whole from palm landmarks instead of driving individual finger bones.
The right hand is intentionally ignored in this MVP. The available right-hand models are static in the current assets and are not mapped to MediaPipe bones yet.
## Known Limitations
@@ -118,3 +126,4 @@ The hand tracking debug panel is a compact HTML grid outside the canvas. `Model
- The virtual hit zone is an approximation based on multiple raycasts, not a real 3D collider.
- There is no smoothing layer for hand position or depth yet.
- The hand visualization is an SVG landmark wireframe.
- The left glove follows the palm as a whole; finger-by-finger bone animation still requires a verified landmark-to-bone mapping and smoothing.