feat add left hand tracking glove model

This commit is contained in:
Tom Boullay
2026-05-02 00:14:56 +02:00
parent 2aa662669f
commit 8fbb2e9428
8 changed files with 156 additions and 6 deletions
+3 -1
View File
@@ -41,7 +41,8 @@ This document describes the code that exists today in the repository.
- `src/components/debug/DebugPerf.tsx` lazily mounts `r3f-perf` in debug mode.
- `src/components/ui/debug/DebugOverlayLayout.tsx` mounts the compact HTML debug overlay when enabled from `lil-gui`.
- `src/components/ui/debug/GameStateDebugPanel.tsx` exposes current game state, main/sub-state switching, previous/next step controls, and reset.
- `src/components/ui/debug/HandTrackingDebugPanel.tsx` shows hand tracking status, usage, model-loaded placeholder, hand count, and fist state while hand tracking is active.
- `src/components/ui/debug/HandTrackingDebugPanel.tsx` shows hand tracking status, usage, loaded glove model, hand count, and fist state while hand tracking is active.
- `src/components/three/handTracking/HandTrackingLeftGlove.tsx` places the rigged `gant_l` model on the detected left hand in the debug physics scene.
- `src/components/debug/scene/DebugHelpers.tsx` mounts debug helpers.
- `src/components/debug/scene/DebugCameraControls.tsx` mounts the free debug camera.
- `lil-gui` global debug controls include camera mode, scene mode, `R3F Perf`, and `Debug Overlay`; interaction-specific controls live in the `Interaction` folder.
@@ -50,6 +51,7 @@ This document describes the code that exists today in the repository.
- `src/components/three/models/` contains reusable model helpers such as `ExplodableModel`.
- `src/components/three/interaction/` contains reusable interaction wrappers such as `InteractableObject`, `TriggerObject`, and `GrabbableObject`.
- `src/components/three/handTracking/` contains R3F hand tracking debug models such as the left glove overlay.
- `src/components/three/gameplay/` contains the current core repair gameplay prototype: the repair case, repair game zone, and module slots.
- `src/components/three/world/` contains reusable world/environment objects such as `SkyModel`.
+11 -2
View File
@@ -16,6 +16,7 @@ The feature is scoped to the debug physics scene rather than production gameplay
4. The backend returns hand data including landmarks, handedness, score, center point, and `isFist`.
5. React stores the latest snapshot in the hand tracking provider.
6. `GrabbableObject` reads that snapshot each frame and uses fist state plus raycasting to grab objects.
7. `HandTrackingLeftGlove` reads the same snapshot and places the rigged `gant_l` model on the detected left hand in the debug physics scene.
## Activation Rules
@@ -104,12 +105,19 @@ The final hold distance is clamped between the configured grab minimum and maxim
The current debug UI includes:
- `HandTrackingDebugPanel` inside `DebugOverlayLayout` for status, usage, model-loaded placeholder, server state, hand count, and fist state
- `HandTrackingDebugPanel` inside `DebugOverlayLayout` for status, usage, loaded glove model, server state, hand count, and fist state
- `HandTrackingVisualizer` for the SVG landmark wireframe
- `HandTrackingLeftGlove` for the left-hand `gant_l` model in the R3F scene
- `r3f-perf` for render performance
- `lil-gui` for scene, camera, lighting, interaction, and grab controls
The hand tracking debug panel is a compact HTML grid outside the canvas. `Model loaded` is currently hardcoded to `none` until model-loading information is wired into the hand tracking flow. The hand wireframe is also HTML/SVG, not a 3D hand model.
The hand tracking debug panel is a compact HTML grid outside the canvas. `Model loaded` displays `gant_l` when a left hand is detected, otherwise `none`. The hand wireframe is also HTML/SVG, not a 3D hand model.
## Left Glove Model
The current left glove MVP uses `public/models/gant_l/model.gltf`, which contains a GLTF skin and armature. For now the model is positioned, oriented, and scaled as a whole from palm landmarks instead of driving individual finger bones.
The right hand is intentionally ignored in this MVP. The available right-hand models are static in the current assets and are not mapped to MediaPipe bones yet.
## Known Limitations
@@ -118,3 +126,4 @@ The hand tracking debug panel is a compact HTML grid outside the canvas. `Model
- The virtual hit zone is an approximation based on multiple raycasts, not a real 3D collider.
- There is no smoothing layer for hand position or depth yet.
- The hand visualization is an SVG landmark wireframe.
- The left glove follows the palm as a whole; finger-by-finger bone animation still requires a verified landmark-to-bone mapping and smoothing.