32 Commits

Author SHA1 Message Date
Tom Boullay d29b01e398 feat(repair): broken parts spawn from exploded model node positions
🔍 Lint / 🪄 Check lint (push) Has been cancelled
🔍 Lint / 🎨 Check format (push) Has been cancelled
🔍 Lint / 🔎 Typecheck (push) Has been cancelled
📊 Quality / 🔒 Security Audit (push) Has been cancelled
📊 Quality / 📋 Dependency Freshness (push) Has been cancelled
📊 Quality / 📦 Bundle Size (push) Has been cancelled
🔍 Lint / 🏗 Build (push) Has been cancelled
Render the exploded mission model during the repairing step so broken
nodes (e.g. ebike refroidisseur) stay visible in the world, and surface
their world positions to RepairRepairingStep so broken pieces spawn
from where they belong on the model rather than from a static offset.

- ExplodableModel: add hideNodeNames (mesh visibility off, restored on
  unmount) and nodeAnchorNames + onNodeAnchorsChange (per-frame world
  positions debounced via signature so React state updates only when
  the values actually move).
- RepairGame: render ExplodableModel during 'repairing' with the broken
  node names hidden + anchors forwarded; threads brokenAnchors state
  to RepairRepairingStep.
- RepairScanSequence + RepairRepairingStep: propagate targetNodeName
  through scanned and fallback broken-part lists.
- RepairRepairingStep: broken parts spawn at brokenAnchors[targetNodeName]
  when set, falling back to legacy BROKEN_PART_START_OFFSETS otherwise.
2026-06-02 19:12:38 +02:00
Tom Boullay 6edc5f7972 docs: refresh hand-tracking notes and drop context-lost investigation 2026-06-02 19:06:32 +02:00
Tom Boullay ae35eb1dfb feat(handtracking): restyle svg visualizer and add silhouette fallback 2026-06-02 19:05:39 +02:00
Tom Boullay 4de86f4e58 feat(repair): align mission data with new pylone glb and broken-part workflow
- Update pylone path references from /models/pylone/model.gltf to .glb
  (galleryModels, mapInstancingConfig, repairMissions). The new glb
  exports a single cable node 'cable2' (no cable1).
- Pylon brokenParts: drop lampe and panneau2; the pylon design no longer
  has detached broken pieces (the cable is just missing from the model).
- Pylon replacement cables target the pylon's 'cable2' node world position
  via targetNodeName (used by the upcoming broken-anchor wiring).
- Ebike refroidisseur replacement and brokenPart both target the bike's
  'refroidisseur' node so the broken piece spawns from its original
  location and the replacement can install in the same slot.
2026-06-02 19:01:47 +02:00
Tom Boullay 5b123f9704 feat(repair): soft-lock mutually exclusive replacement parts
When a replacement part with a caseLockGroup is grabbed, sibling parts
sharing the same group become non-interactable and ghosted (35% opacity)
until the held part is released. This implements the pylon cable choice
where the player picks either cable1 or cable2 (both valid) without
being able to grab both simultaneously.

- GrabbableObject: add disabled prop (skips interaction frame logic and
  unmounts InteractableObject so it does not register with the manager)
  and onGrabChange callback fired on press, release, hand grab, and hand
  release. Force-releases when disabled becomes true mid-grab.
- SimpleModel: add opacity prop, traversed onto cloned mesh materials
  (safe because cloneResources clones materials per instance).
- RepairObjectModel: forward ghosted prop as opacity 0.35.
- RepairRepairingStep: track heldPartByLockGroup and pass disabled +
  ghosted to siblings of the currently held part.
2026-06-02 18:46:34 +02:00
Tom Boullay d1bf438465 feat(repair): inject ebike + pylon parts at packderelance anchors
- Ebike replacement parts: cooling core (correct, anchored at refroidisseur)
  + four distractors anchored at cabledroit/cablegauche/pucehaut/pucebas.
  Removes the ad-hoc gant_l/talkie distractors in favor of consistent
  case-anchored visuals.
- Pylon replacement parts: cable1 + cable2 (alternative correct, both with
  caseLockGroup 'pylon-cable' for upcoming soft-lock) + refroidisseur and
  two puce distractors anchored to packderelance.
- Farm replacement parts kept as-is (caseAnchor undefined falls back to
  placeholder slot positions for backward compatibility).
- RepairGame threads anchors from RepairCaseModel through RepairMissionCase
  to RepairRepairingStep; replacement-part initial position now resolves
  to the anchor world position when caseAnchor is set, falling back to the
  legacy slot index otherwise.
2026-06-02 18:37:12 +02:00
Tom Boullay d2ce990165 feat(repair): support multiple required parts and per-part case anchor
- RepairMissionConfig.requiredReplacementPartId (string) is replaced by
  requiredReplacementPartIds (readonly string[]) so a mission can accept
  several alternative correct parts (e.g. pylon will accept either cable).
- RepairMissionPartConfig gains optional caseAnchor (where the standalone
  spawns inside packderelance), caseLockGroup (mutually exclusive parts),
  and targetNodeName (snap onto a node of the broken model rather than a
  placeholder slot in the case).
- RepairScannedBrokenPart gains targetNodeName so scan results can carry
  this hint through to the repairing step.
- RepairRepairingStep validation logic (placed/wrong/feedback) now matches
  any id in requiredReplacementPartIds. Existing data is migrated mechanically
  (single-element arrays); part-level new fields are wired in subsequent
  commits.
2026-06-02 18:26:45 +02:00
Tom Boullay 7d2a257e84 feat(repair): expose case part anchors and fix lid node name
- Fix REPAIR_CASE_LID_NODE_NAME from "partiesup" to "partsup" (the actual
  node name in packderelance.gltf), restoring the lid open/close animation
  that was silently no-oping since introduction.
- Add REPAIR_CASE_PART_ANCHOR_NAMES (cabledroit, cablegauche, pucehaut,
  pucebas, refroidisseur) and REPAIR_CASE_PART_ANCHOR_FALLBACKS for
  case-local positions used when the GLTF lacks a node (refroidisseur).
- RepairCaseModel now resolves these anchor nodes on mount, hides existing
  meshes underneath them, and creates lightweight Object3D placeholders for
  missing names so the anchoring pipeline is uniform.
- Each frame, anchor world positions are converted to step-local space and
  emitted via the new onAnchorsChange callback (debounced via signature like
  placeholders). Consumers added in subsequent commits.
2026-06-02 18:20:43 +02:00
Tom Boullay 58eb60292f update: model pylone 2026-06-02 18:01:14 +02:00
Tom Boullay 73c6d7d50d fix(handtracking): bump browser camera to 640x480 for detection
The browser MediaPipe model (hand_landmarker.task float16) was failing
to detect hands at 320x240 — MediaPipe ran for 10s straight without
ever returning a hand. Bumping the requested camera resolution to
640x480 (browser only — backend still ships 320x240 JPEGs over the
WebSocket) makes the model reliably pick hands up.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-06-02 17:39:43 +02:00
Tom Boullay d9cacdad12 fix(handtracking): stabilize provider root and linger enabled
(1) HandTrackingProvider always renders the same JSX root
(HandTrackingRuntime) so toggling `enabled` no longer remounts the
<Canvas> below — that remount was destroying the WebGL context every
time the player entered an interaction zone.

(2) Add HAND_TRACKING_LINGER_MS (2s) cooldown on `enabled` so brief
walk-throughs of a trigger zone don't tear down MediaPipe before it
has time to initialize the webcam + model + first frame (cold start
~800ms).

Resolves the WebGL context lost + respawn loop and restores visible
hand tracking in the backend runtime. Browser JS runtime detection
quality is a separate follow-up.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-06-02 17:27:04 +02:00
Tom Boullay ab88ab722f Update intro.mp4 2026-06-02 17:24:56 +02:00
Tom Boullay a30a9a2d29 fix(handtracking): absorb React StrictMode double-mount
🔍 Lint / 🪄 Check lint (push) Has been cancelled
🔍 Lint / 🎨 Check format (push) Has been cancelled
🔍 Lint / 🔎 Typecheck (push) Has been cancelled
📊 Quality / 🔒 Security Audit (push) Has been cancelled
📊 Quality / 📋 Dependency Freshness (push) Has been cancelled
📊 Quality / 📦 Bundle Size (push) Has been cancelled
🔍 Lint / 🏗 Build (push) Has been cancelled
In dev, <StrictMode> intentionally mounts → unmounts → remounts each
effect to surface non-idempotent code. The hand tracking hooks were
calling getUserMedia and creating MediaPipe / WebSocket runtimes on
every mount, which in practice ran the full start/stop/start cycle
inside a few milliseconds and pushed WebGL over its limit on top
of the loaded scene → context lost.

Add HAND_TRACKING_RUNTIME_START_DELAY_MS (80ms) and delay the actual
start() call behind a setTimeout in both useBrowserHandTracking and
useRemoteHandTracking. The cleanup clears the timer, so a fast
mount/unmount never reaches start(). 80ms is invisible to the user
(<5 frames at 60fps) and also absorbs rapid `nearby` toggles at
trigger borders.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-06-02 16:54:28 +02:00
Tom Boullay d217c3376b fix(handtracking): reduce GPU pressure on WebGL context loss
Several mitigations against the WebGL context lost that fires when
hand tracking starts on a loaded scene:

- Canvas: fixed DPR [1,1], antialias off, scoped id="game-canvas",
  context-lost handler releases MediaPipe and logs GPU memory counters
- optimizeGLTFScene: cap anisotropy at 2 and stop forcing mipmaps /
  needsUpdate on every pass — avoids massive texture re-uploads
- MediaPipe: force CPU delegate (HAND_TRACKING_BROWSER_DELEGATE),
  cache the landmarker instance, and expose releaseBrowserHandLandmarker
- useBrowserHandTracking / useRemoteHandTracking: idempotent cleanup
  guarded by a cleanedUp flag, try/catch around the detect loop, and
  release of the landmarker on stop
- World: mount HandTrackingGlove only when the matching hand is
  actually present in the snapshot (status connected + hands.length > 0)
- HandTrackingGlove: drop the eager useGLTF.preload that was running
  at startup whether or not hand tracking was used

Does not yet absorb the React StrictMode double-mount — that is the
follow-up commit.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-06-02 16:48:39 +02:00
Tom Boullay 864e075b42 chore(debug): keep hand tracking source controller display in sync
Store the lil-gui Source controller so setHandTrackingSource() from
the settings menu can refresh its display, and log the transition
for traceability.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-06-02 16:34:05 +02:00
Tom Boullay 3fe5b32de2 fix(player): scope pointer lock and ground snap to game scene
PointerLockControls now targets #game-canvas and respects the
settings menu, and document.exitPointerLock() only runs when a
pointer lock is actually active. The terrain ground snap in
PlayerController is gated on sceneMode === "game" so it doesn't
fight the physics test scene's flat floor.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-06-02 16:33:54 +02:00
Tom Boullay 72cb9f5be6 update: ebike model 2026-06-02 16:33:42 +02:00
Tom Boullay f708c4cd2e chore(debug): tune physics test scene and drop noisy waypoint logs
Lower grabbable spawn, expose GPS preview position/rotation as
constants, fix physics spawn to use PLAYER_EYE_HEIGHT, and silence
the console.log noise around waypoint loading in TestMap.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-06-02 16:33:39 +02:00
Tom Boullay c61760dafd docs: document webgl context lost investigation
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-06-02 16:02:58 +02:00
Tom Boullay 18dd2ae49d fix(electricienne): remove alphaMode BLEND + doubleSided causing face artifacts
Solid character mesh does not need transparency; BLEND combined with doubleSided
caused depth-sort issues rendering inside-faces over front-faces (visible as
triangular orange artifacts on the face). Default OPAQUE single-sided is correct.
2026-06-02 15:36:43 +02:00
Tom Boullay def0609383 chore(talkie): consolidate to single GLB, remove orphan LOD assets
- Replace public/models/talkie/* (PNG textures + .bin + .gltf) with single model.glb
- Delete unused public/models/talkie-LOD/ folder (no longer referenced in MAP_LOD_MODEL_PATHS)
- Remove farm-radio-distractor (talkie) from farm repair mission distractors
2026-06-02 15:36:20 +02:00
Tom Boullay 19a1d20a97 fix(repair): remove talkie LOD + drop talkie distractor from ebike
- Talkie does not need a LOD swap; remove the entry from
  MAP_LOD_MODEL_PATHS so the same model.glb is used at any distance.
- The ebike repair distractors are meant to be other disassembled bike
  parts, not random props. Drop the talkie radio distractor and keep
  only the (thematically plausible) insulation glove alongside the
  correct cooling core replacement.
2026-06-02 15:27:30 +02:00
Tom Boullay 49ebacfbfb chore(talkie): point full-detail talkie path at model.glb
The talkie folder now ships a single binary GLB; update the four call
sites (TalkieModel, gallery, two repair mission entries) to load
model.glb. The talkie-LOD path is unchanged.
2026-06-02 15:06:44 +02:00
Tom Boullay 68253fae41 Update lightingConfig.ts
🔍 Lint / 🪄 Check lint (push) Has been cancelled
🔍 Lint / 🎨 Check format (push) Has been cancelled
🔍 Lint / 🔎 Typecheck (push) Has been cancelled
📊 Quality / 🔒 Security Audit (push) Has been cancelled
📊 Quality / 📋 Dependency Freshness (push) Has been cancelled
📊 Quality / 📦 Bundle Size (push) Has been cancelled
🔍 Lint / 🏗 Build (push) Has been cancelled
2026-06-02 14:52:54 +02:00
Tom Boullay 2dabb73d3d fix(vegetation): scale-correct sapin and buisson LOD meshes
The exported sapin-LOD and buisson-LOD models do not match their full
detail counterpart's baseline scale, so they appear oversized once the
LOD swap kicks in. Add MAP_LOD_SCALE_MULTIPLIERS keyed by map name
(sapin: 0.5, buisson: 0.8) and apply it on top of the chunk's existing
scaleMultiplier whenever the resolved model path is the LOD variant.
2026-06-02 14:50:07 +02:00
Tom Boullay 4f1b3b4ff3 fix(graphics): tune presets, single-line ui, vegetation LOD by nearest instance
🔍 Lint / 🪄 Check lint (push) Has been cancelled
🔍 Lint / 🎨 Check format (push) Has been cancelled
🔍 Lint / 🔎 Typecheck (push) Has been cancelled
📊 Quality / 🔒 Security Audit (push) Has been cancelled
📊 Quality / 📋 Dependency Freshness (push) Has been cancelled
📊 Quality / 📦 Bundle Size (push) Has been cancelled
🔍 Lint / 🏗 Build (push) Has been cancelled
- Bump high to 30m chunk / HD<20m and ultra to HD<30m so HD models
  persist further before swapping.
- Render the 5 graphics preset cards on a single row.
- Vegetation LOD selection now uses the distance to the nearest instance
  in each chunk instead of the chunk centre, matching MapInstancingSystem
  and avoiding premature LOD swaps when the camera enters a chunk.
2026-06-02 14:33:16 +02:00
Tom Boullay 627c8d4eb9 feat(vegetation): wire arbre/sapin/buisson into LOD system
Register the three new vegetation LOD models in MAP_LOD_MODEL_PATHS and
extend VegetationSystem with per-chunk distance-based LOD selection
(mirroring MapInstancingSystem). Chunk model paths are re-evaluated on
the existing CHUNK_CONFIG.updateInterval cadence and Suspense keys
include the resolved path so a swap unmounts the previous instanced mesh
cleanly.
2026-06-02 14:17:19 +02:00
Tom Boullay 0b801888f0 update: fix name assets + add LOD 2026-06-02 14:13:55 +02:00
Tom Boullay a180b89ee6 fix(pylone): convert lampe BLEND to MASK in GLTFs
Switching alphaMode from BLEND to MASK with a 0.5 cutoff lets the lampe
material participate in instanced shadow rendering correctly. BLEND is
known to interact badly with InstancedMesh shadow casting, producing
incorrect or missing shadows on the pylone lampe.
2026-06-02 13:52:08 +02:00
Tom Boullay 3e66e31117 feat(graphics): add max preset (no chunk streaming, LOD@50m)
Restore ultra to its original behaviour (50m chunk streaming, HD within
20m, no fog) and introduce a new max preset that disables chunk streaming
entirely (loads all chunks unconditionally) and pushes the HD/LOD swap
distance to 50m. Add chunkStreamingEnabled flag to GraphicsPresetConfig
so the streaming gate honours the preset. The settings card label shows
'All' when streaming is off.
2026-06-02 13:51:33 +02:00
Tom Boullay 2c194cdd2e fix: use player ebike speed
🔍 Lint / 🪄 Check lint (push) Has been cancelled
🔍 Lint / 🎨 Check format (push) Has been cancelled
🔍 Lint / 🔎 Typecheck (push) Has been cancelled
📊 Quality / 🔒 Security Audit (push) Has been cancelled
📊 Quality / 📋 Dependency Freshness (push) Has been cancelled
📊 Quality / 📦 Bundle Size (push) Has been cancelled
🔍 Lint / 🏗 Build (push) Has been cancelled
2026-06-02 09:48:44 +02:00
Tom Boullay feaf502e44 feat: support webm mission notifications 2026-06-02 09:40:18 +02:00
321 changed files with 1352 additions and 1023 deletions
+68 -20
View File
@@ -10,17 +10,23 @@ It is now also available to the production repair flow when a mission reaches a
## Runtime Flow ## Runtime Flow
1. The browser captures webcam frames in `src/hooks/handTracking/useRemoteHandTracking.ts`. The frontend can run hand tracking with two interchangeable sources, selected from the debug source controller:
2. Frames are sent to the local Python backend over WebSocket.
3. The backend runs MediaPipe hand landmark detection. - **Browser JS** (`src/hooks/handTracking/useBrowserHandTracking.ts`) runs MediaPipe `hand_landmarker.task` directly in the browser via `@mediapipe/tasks-vision`. Default for debug.
4. The backend returns hand data including landmarks, handedness, score, center point, and `isFist`. - **Backend** (`src/hooks/handTracking/useRemoteHandTracking.ts`) sends webcam frames as JPEG over WebSocket to a local Python process that runs MediaPipe and returns landmarks.
5. React stores the latest snapshot in the hand tracking provider.
6. `GrabbableObject` reads that snapshot each frame and uses fist state plus raycasting to grab objects. Both sources funnel into the same `HandTrackingContext` so all consumers see one shared snapshot:
7. `HandTrackingGlove` reads the same snapshot and places the rigged `gant_l` and `gant_r` models on the detected hands when hand tracking is active.
1. The active source captures or receives landmarks.
2. The hook applies an EMA smoothing pass on the landmarks before publishing the snapshot.
3. `HandTrackingProvider` exposes that snapshot through React context.
4. `GrabbableObject` reads the snapshot each frame and uses the fist state plus raycasting to grab objects.
5. `HandTrackingGlove` reads the same snapshot and places a rigged glove on each detected hand.
6. `HandTrackingVisualizer` paints an SVG wireframe overlay on top of the canvas.
## Activation Rules ## Activation Rules
Hand tracking is intentionally gated so the webcam and backend are not used all the time. Hand tracking is gated so the webcam and runtime are only spun up when actually needed.
The debug activation conditions are: The debug activation conditions are:
@@ -28,16 +34,26 @@ The debug activation conditions are:
- scene mode is `physics` - scene mode is `physics`
- the player is near an interaction, is holding an object, or is hand-holding an object - the player is near an interaction, is holding an object, or is hand-holding an object
This keeps hand tracking active while the player is inside an interaction zone, even if the camera is not aimed directly at the object.
The production repair activation conditions are: The production repair activation conditions are:
- active `mainState` is `ebike`, `pylon`, or `farm` - active `mainState` is `ebike`, `pylon`, or `farm`
- the active mission step is `inspected`, `repairing`, `reassembling`, or `done` - the active mission step is `inspected`, `repairing`, `reassembling`, or `done`
This keeps the webcam off during `waiting`, `fragmented`, and `scanning`, then enables hand input only when the repair flow is expected to use hands. This keeps the webcam off during `waiting`, `fragmented`, and `scanning`.
In the current production repair flow, `inspected` uses a two-fists hold gesture to advance to `fragmented`. The hold must last one second and is independent from local object interaction distance once the mission is in the correct state. Keyboard input for the same transition is handled separately by the repair case trigger, so pressing `E` requires the case to be focused through the shared interaction system. ### Linger
Once activation turns off (player walks back out of a trigger zone, or a mission step transitions away), the runtime stays alive for `HAND_TRACKING_LINGER_MS` (2000 ms) before being torn down. This gives MediaPipe enough time to finish initializing the webcam and load the model on a fresh entry — without the linger, a quick walk-through of a trigger zone never produces a detected hand.
## Provider Stability
`HandTrackingProvider` always renders the same JSX root (`HandTrackingRuntime`) and exposes `enabled` as a prop. Returning two different element types (`<HandTrackingContext value=IDLE>` vs `<ActiveHandTrackingProvider>`) used to be the historical shape and was the root cause of WebGL context loss: every `enabled` toggle forced React to remount the entire subtree, including the `<Canvas>`, which destroyed the WebGL renderer.
The two source hooks are therefore mounted in permanence with an `enabled` flag that they early-return on. No webcam or MediaPipe resources are created while `enabled` is false.
## StrictMode Resilience
In development, `<StrictMode>` mounts → unmounts → remounts each effect to surface non-idempotent code. The two source hooks delay their actual `start()` call by `HAND_TRACKING_RUNTIME_START_DELAY_MS` (80 ms) and clear the timer on cleanup, so a StrictMode double-mount or a rapid `nearby` flicker never reaches `getUserMedia` twice.
## Backend ## Backend
@@ -52,7 +68,27 @@ The Python process uses MediaPipe and the local model file:
backend/hand_landmarker.task backend/hand_landmarker.task
``` ```
The backend sends normalized hand coordinates and landmarks. The frontend treats the values as screen-space inputs, then maps them into world space with the active Three.js camera. The frontend sends JPEG frames at `HAND_TRACKING_FRAME_WIDTH × HAND_TRACKING_FRAME_HEIGHT` (320×240) to keep WebSocket bandwidth low. The backend sends normalized hand coordinates and landmarks.
## Browser MediaPipe
The browser path uses `hand_landmarker.task` (float16) downloaded from Google's MediaPipe model storage. The requested webcam resolution is **640×480** (`HAND_TRACKING_BROWSER_CAMERA_WIDTH/HEIGHT`), independent from the backend's 320×240. The float16 model is more sensitive than the backend Python model and needs the higher-resolution frame to detect hands reliably.
The MediaPipe delegate is currently `"GPU"`. CPU works too but is significantly slower; on a loaded scene the inference drops to ~5fps and the user feels noticeable lag during grab. MediaPipe creates its own WebGL context separate from Three.js, so there is no direct contention.
A singleton instance of `HandLandmarker` is cached in `src/lib/handTracking/browserHandTracking.ts`. `releaseBrowserHandLandmarker()` is called on cleanup and on WebGL context lost.
## Smoothing
MediaPipe at ~10 fps produces noticeable landmark jitter that, when fed raw into the scene, makes both the glove rig and any grabbed object tremble.
A simple exponential moving average is applied to every landmark before the snapshot is published:
```ts
smoothed.x = previous.x * (1 - factor) + next.x * factor;
```
The factor is `HAND_TRACKING_LANDMARK_SMOOTHING` (0.4). Hands are matched across frames by `handedness` so left/right don't bleed into each other.
## Frontend Data Shape ## Frontend Data Shape
@@ -106,24 +142,36 @@ This is less expressive than true depth-aware hand movement, but it is more stab
The current debug UI includes: The current debug UI includes:
- `HandTrackingDebugPanel` inside `DebugOverlayLayout` for status, usage, loaded glove model, server state, hand count, and fist state - `HandTrackingDebugPanel` inside `DebugOverlayLayout` for status, usage, loaded glove model, server state, hand count, and fist state
- `HandTrackingVisualizer` for the SVG landmark wireframe fallback - `HandTrackingVisualizer` for the SVG landmark overlay
- `HandTrackingGlove` for the left-hand `gant_l` and right-hand `gant_r` models in the R3F scene - `HandTrackingFallback` for the last-resort hand silhouette overlay
- `HandTrackingGlove` for the per-hand rigged glove models in the R3F scene
- `r3f-perf` for render performance - `r3f-perf` for render performance
- `lil-gui` for scene, camera, lighting, interaction, and grab controls - `lil-gui` for scene, camera, lighting, interaction, and grab controls
The hand tracking debug panel is a compact HTML grid outside the canvas. `Model loaded` displays the successfully loaded glove models. The SVG hand wireframe is only a fallback while models are loading or if a glove model fails to load. The SVG visualizer uses a "blueish hand" style: white connection lines between landmarks, cyan circles with a dark blue outline. The outline gets thicker when the hand is detected as a fist, so the user gets a visual confirmation of the grab gesture without having to look at the debug panel.
The fallback overlay (`HandTrackingFallback`) draws a simple open-hand or fist silhouette positioned on the detected wrist landmark. It only renders for a hand whose matching glove is in the `"error"` state in `useHandTrackingGloveStatus`. This guarantees the user always sees something on their hand even when the 3D glove model fails to load.
## Glove Models ## Glove Models
The current glove MVP uses `public/models/gant_l/model.gltf` and `public/models/gant_r/model.gltf`, which contain GLTF skins and armatures. Each model is positioned, oriented, and scaled from palm landmarks, then each finger bone chain is rotated toward the matching MediaPipe landmark chain. `HandTrackingGlove` loads `public/models/gant_l/model.gltf` for both hands. The right hand applies `scale.x = -1` at the group level to mirror the mesh, so the thumb ends up on the correct side. Both hands therefore share the same rig and the same material.
The glove models are intentionally smaller than the raw SVG overlay so they do not dominate the camera view. The historical `public/models/gant_r/model.gltf` is kept as legacy but is not loaded by the frontend — its GLB embeds three skeletons (`Hand_l`, `Hand_l_pad`, `Hand_r`) plus a `galet` mesh, which made the finger rig unreliable.
The `gant_l` material is set to `alphaMode: OPAQUE` with `doubleSided: true`. The opaque mode prevents transparency sorting issues that made folded fingers disappear behind the palm; the double-sided flag covers the back faces revealed by the mirror scale on the right hand.
Two additional glove variants exist on disk:
- `public/models/gant_l_pad/model.gltf`
- `public/models/gant_r_pad/model.gltf`
They are intended for future swap-by-state usage but are **not yet rigged**. They cannot be animated by MediaPipe landmarks in their current form — re-exporting them from Blender with the same armature structure as `gant_l` is a prerequisite.
## Known Limitations ## Known Limitations
- Production usage is currently limited to repair mission steps that explicitly need hands. - Production usage is currently limited to repair mission steps that explicitly need hands.
- MediaPipe depth is relative and currently not used for stable object depth control. - MediaPipe depth is relative and currently not used for stable object depth control.
- The virtual hit zone is an approximation based on multiple raycasts, not a real 3D collider. - The virtual hit zone is an approximation based on multiple raycasts, not a real 3D collider.
- There is no smoothing layer for hand position or depth yet. - The right glove is a mirrored copy of `gant_l` rather than its own mesh; in the future a dedicated right-hand model would give a better visual.
- The SVG hand visualization is a fallback, not the primary display when glove models load correctly. - The `_pad` glove variants are not rigged yet, so swap-by-state (normal ↔ pad) is not wired in.
- Finger bone animation is an approximate landmark-to-bone mapping; it still needs calibration for per-model twist, offsets, and smoothing. - Finger bone animation is an approximate landmark-to-bone mapping; it still needs calibration for per-model twist, offsets, and smoothing.
LFS Regular → Executable
BIN
View File
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.

Some files were not shown because too many files have changed in this diff Show More