merge: sync develop into env manager
🔍 Lint / 🪄 Check lint (pull_request) Has been cancelled
🔍 Lint / 🎨 Check format (pull_request) Has been cancelled
🔍 Lint / 🔎 Typecheck (pull_request) Has been cancelled
🔍 Lint / 🏗 Build (pull_request) Has been cancelled
📊 Quality / 🔒 Security Audit (pull_request) Has been cancelled
📊 Quality / 📋 Dependency Freshness (pull_request) Has been cancelled
📊 Quality / 📦 Bundle Size (pull_request) Has been cancelled

This commit is contained in:
Tom Boullay
2026-05-12 10:56:56 +02:00
171 changed files with 4560 additions and 1507 deletions
+2 -3
View File
@@ -37,9 +37,8 @@ Use `useClonedObject` when a GLTF scene is reused by a component instance. It me
src/components/three/
├── gameplay/
│ ├── RepairCaseModel.tsx
│ ├── RepairCaseObject.tsx
── RepairGameZone.tsx
│ └── RepairModuleSlot.tsx
│ ├── RepairGame.tsx
── RepairRepairingStep.tsx
├── interaction/
│ ├── GrabbableObject.tsx
│ ├── InteractableObject.tsx
+25 -7
View File
@@ -14,10 +14,24 @@ This document describes the code that exists today in the repository.
- debug helpers and debug camera mode
- either the map scene or the debug physics test scene
- the player rig when the active camera mode is `player`
- `src/world/GameMap.tsx` loads map nodes from `public/map.json`, resolves available models, and builds the collision octree.
- `src/world/debug/TestMap.tsx` provides a debug-oriented interaction and physics map.
- `src/hooks/world/useWorldSceneLoading.ts` owns the production scene loading state shared by `World`, `GameMap`, and the player octree readiness.
- `src/world/GameMap.tsx` loads map nodes from `public/map.json`, resolves available models, renders them progressively, and shows fallback cubes for missing models.
- `src/world/GameMapCollision.tsx` builds the player collision octree from dedicated collision nodes only.
- `src/world/GameStageContent.tsx` is wrapped in Rapier `Physics` in the production game scene so stage gameplay objects can use physics without moving the map or player to Rapier. It now mounts reusable `RepairGame` instances for `bike`, `pylone`, and `ferme` mission states.
- `src/world/debug/TestMap.tsx` provides a debug-oriented interaction and physics map with the existing grab/trigger/model-preview objects plus separate `Bike`, `Pylone`, and `Farm` repair playground zones.
- `src/world/player/Player.tsx` mounts the camera and controller.
- `src/world/player/PlayerController.tsx` owns pointer lock movement, jump handling, and interaction input.
- `src/world/player/PlayerController.tsx` owns pointer lock movement, jump handling, repair-step movement locking, and interaction input.
## Physics Boundaries
The project currently uses two collision layers with separate responsibilities:
- `GameMapCollision` builds an octree used by the player controller for map collision.
- The player octree must be built from a small collision-only subset of map nodes. It currently uses the `terrain` node only instead of traversing the full visible map, because building an octree from all rendered props can overload the browser renderer.
- `GameStageContent` is wrapped in Rapier `Physics` for gameplay objects such as repair triggers, cases, grabbables, and future mission-specific objects.
- `TestMap` owns its own Rapier `Physics` playground so repair gameplay can be tuned per mission state without depending on the production map layout.
Keep the player and map octree outside the Rapier provider until there is a deliberate migration plan. This avoids mixing player movement rules with object physics before the gameplay systems need it.
## Interaction Model
@@ -33,6 +47,7 @@ This document describes the code that exists today in the repository.
- `src/managers/AudioManager.ts` provides pooled one-shot playback, looped music playback, category volumes, and optional stereo pan for one-shot sounds.
- Supported audio categories are `music`, `sfx`, and `dialogue`.
- Trigger interactions may play SFX directly through `AudioManager`.
- Detailed audio documentation lives in `docs/technical/audio.md`.
## Settings Menu
@@ -78,6 +93,7 @@ This document describes the code that exists today in the repository.
- `src/components/ui/debug/DebugOverlayLayout.tsx` mounts the compact HTML debug overlay when enabled from `lil-gui`.
- `src/components/ui/debug/GameStateDebugPanel.tsx` exposes current game state, main/sub-state switching, previous/next step controls, and reset.
- `src/components/ui/debug/HandTrackingDebugPanel.tsx` shows hand tracking status, usage, loaded glove model, hand count, and fist state while hand tracking is active.
- `src/components/ui/SceneLoadingOverlay.tsx` displays the fullscreen loading state for 3D scenes, including the production game scene, debug physics scene, and editor scene.
- `src/components/three/handTracking/HandTrackingGlove.tsx` places the rigged `gant_l` and `gant_r` models on detected hands in the debug physics scene.
- `src/components/debug/scene/DebugHelpers.tsx` mounts debug helpers.
- `src/components/debug/scene/DebugCameraControls.tsx` mounts the free debug camera.
@@ -88,7 +104,7 @@ This document describes the code that exists today in the repository.
- `src/components/three/models/` contains reusable model helpers such as `ExplodableModel`.
- `src/components/three/interaction/` contains reusable interaction wrappers such as `InteractableObject`, `TriggerObject`, and `GrabbableObject`.
- `src/components/three/handTracking/` contains R3F hand tracking debug models such as the glove overlays.
- `src/components/three/gameplay/` contains the current core repair gameplay prototype: the repair case, repair game zone, and module slots.
- `src/components/three/gameplay/` contains the reusable production `RepairGame` flow, repair case, repair steps, and repair prompt components.
- `src/components/three/world/` contains reusable world/environment objects such as `SkyModel`.
## Editor System
@@ -106,20 +122,22 @@ This document describes the code that exists today in the repository.
- `src/utils/editor/loadEditorScene.ts` handles editor-only folder upload parsing.
- `src/utils/map/loadMapSceneData.ts` is shared by the game scene and editor to load `public/map.json` and resolve model URLs.
- `src/types/editor/editor.ts` contains the shared `MapNode`, `SceneData`, and `TransformMode` types.
- `src/types/gameplay/repairMission.ts` contains shared repair mission ids, mission steps, and guards used across store, config, debug UI, and gameplay components.
## Map Data
- `public/map.json` is expected to be a `MapNode[]`.
- Each map node `name` maps to `public/models/{name}/model.glb` when available, with `public/models/{name}/model.gltf` kept as fallback.
- The editor renders a fallback cube for missing models.
- The game scene filters out nodes whose model cannot be resolved.
- The game scene renders fallback cubes for nodes whose model cannot be resolved.
- The game scene currently uses `terrain` as the collision source for the player octree. Additional collision nodes should be explicit lightweight collision assets, not arbitrary visible decoration models.
## Current Limitations
- The repository is a prototype, not the full intended game runtime.
- `src/world/debug/TestMap.tsx` is part of the active scene composition.
- There is no central gameplay orchestrator such as `GameManager`.
- Missions and zones are not implemented.
- Dialogue branching and gameplay-triggered orchestration are still limited.
- Mission state exists in Zustand and the repair flow is implemented as a prototype for the current repair missions.
- Cinematics and dialogues exist as prototype timecode-driven systems; dialogue branching and broader gameplay orchestration are still limited.
- The player uses octree collision and simple movement rules, not a complete gameplay physics stack.
- Editor save-to-server is implemented as a Vite dev-server plugin, not a production backend API.
+217
View File
@@ -0,0 +1,217 @@
# Audio Technical Notes
This document describes the audio systems that exist in the current codebase.
## Scope
Audio is currently split into three runtime categories:
- `music`: looped background music
- `dialogue`: spoken dialogue audio linked to subtitles
- `sfx`: one-shot interaction and feedback sounds
The shared runtime service is `src/managers/AudioManager.ts`. User-facing volume settings live in `src/managers/stores/useSettingsStore.ts` and are forwarded to `AudioManager` by category.
## AudioManager
`AudioManager` is a singleton side-effect service. It owns browser audio elements, category volumes, pooled one-shot sounds, music playback, and stereo panning for one-shot sounds.
Supported public methods:
- `playMusic(path, volume)`: starts or updates a looped music track.
- `stopMusic()`: stops the active music track.
- `playSound(path, volume, options)`: plays a pooled one-shot sound and returns its `HTMLAudioElement`.
- `setCategoryVolume(category, volume)`: updates `music`, `sfx`, or `dialogue` volume.
- `getCategoryVolume(category)`: reads the current category volume.
- `destroy()`: stops music, clears pools, closes the audio context, and resets the singleton.
One-shot sounds are pooled by path with a maximum pool size per sound. If every element in a pool is busy, the pool grows until the limit, then recycles an existing element.
Browser autoplay restrictions are handled in `playMusic()`: if playback is blocked by the browser, the manager waits for a user `pointerdown` or `keydown`, then retries the music.
## Music
Runtime music is mounted by `src/world/GameMusic.tsx`.
Current behavior:
- `GameMusic` calls `AudioManager.getInstance().playMusic()` on mount.
- The current music path is `/sounds/musique/test.mp3`.
- The base music volume is `0.33` before category volume is applied.
- On unmount, `GameMusic` calls `stopMusic()`.
Effective music volume is:
```txt
base music volume * settings music volume
```
Use `music` only for long-running looped background tracks. Do not use `playSound()` for music, because one-shot pooling is designed for short overlapping sounds.
## Sound Effects
SFX are short one-shot sounds. They should use `AudioManager.playSound()` with the default category or with `{ category: "sfx" }`.
Example:
```ts
AudioManager.getInstance().playSound("/sounds/sfx/click.mp3", 0.8, {
category: "sfx",
pan: 0,
});
```
Useful options:
- `category`: `sfx` or `dialogue`; defaults to `sfx`.
- `pan`: stereo panning from `-1` left to `1` right.
- `playbackRate`: playback speed multiplier.
SFX volume is controlled by the settings menu through the `sfx` category volume.
## Dialogues
Runtime dialogue data lives under `public/sounds/dialogue/`.
```txt
public/
└── sounds/
└── dialogue/
├── dialogues.json
└── subtitles/
├── fr/
│ ├── narrateur.srt
│ ├── fermier.srt
│ └── electricienne.srt
└── en/
├── narrateur.srt
├── fermier.srt
└── electricienne.srt
```
The dialogue manifest shape is defined in `src/types/dialogues/dialogues.ts`.
Each dialogue entry contains:
- `id`: stable dialogue identifier
- `voice`: voice group, currently `narrateur`, `fermier`, or `electricienne`
- `audio`: runtime audio path
- `subtitleCueIndex`: cue number inside that voice/language SRT file
- `timecode`: optional global trigger time in seconds from scene start
Dialogues are played through `src/utils/dialogues/playDialogue.ts`.
Important functions:
- `playDialogueById(manifest, dialogueId)`: plays a dialogue from an already loaded manifest.
- `queueDialogueById(manifest, dialogueId)`: queues dialogue playback so multiple requests do not overlap.
- `playGameplayDialogueById(dialogueId)`: loads the gameplay manifest once and queues a dialogue by ID.
- `clearQueuedDialogues()`: resolves pending dialogue requests and clears the queue.
Dialogue audio uses `AudioManager.playSound()` with `{ category: "dialogue" }`, so it follows the dialogue volume setting.
## Dialogue And SRT Link
The subtitle model is one SRT file per voice and language, not one SRT file per dialogue.
A dialogue chooses its subtitle by combining:
1. `voice`
2. selected subtitle language from settings
3. `subtitleCueIndex`
For example, this dialogue:
```json
{
"id": "narrateur_bienvenueaaltera",
"voice": "narrateur",
"audio": "/sounds/dialogue/narrateur/bienvenueaaltera.mp3",
"subtitleCueIndex": 1
}
```
loads cue `1` from:
```txt
public/sounds/dialogue/subtitles/fr/narrateur.srt
```
when the subtitle language is French, or from:
```txt
public/sounds/dialogue/subtitles/en/narrateur.srt
```
when the subtitle language is English.
If the selected language is missing, the loader falls back to French. Missing English SRT files are warnings during validation, not runtime errors.
SRT timecodes are relative to the dialogue audio file. They are not relative to the game clock and not relative to a cinematic timeline.
## Subtitle Runtime
`playDialogueById()` loads the matching subtitle cue with `loadDialogueSubtitleCue()` before playing the audio.
While audio plays:
- `timeupdate` checks `audio.currentTime`
- the active subtitle is written to `useSubtitleStore`
- `src/components/ui/Subtitles.tsx` renders the current speaker and text
- `ended` and `pause` clear the subtitle
The subtitle overlay respects settings from `useSettingsStore`, including visibility and selected language.
## Global Timecode Dialogues
`src/world/GameDialogues.tsx` loads the dialogue manifest and triggers entries that define `timecode`.
This is useful for simple global scene timing. It should not be used for dialogue that belongs to a cinematic. Cinematic-owned dialogue should be triggered by `dialogueCues` in `public/cinematics.json` instead, otherwise the same dialogue can play twice.
## Cinematic Dialogue Cues
`public/cinematics.json` can include `dialogueCues`.
Each cue contains:
- `time`: seconds relative to the cinematic start
- `dialogueId`: ID from `dialogues.json`
`src/world/GameCinematics.tsx` uses those cues to play dialogue during camera timelines. This keeps camera movement and dialogue playback synchronized without relying on global scene time.
## Editor Tooling
The `/editor` route provides three audio-related tools:
- `Dialogues`: edits `public/sounds/dialogue/dialogues.json` and previews dialogue playback.
- `SRT`: edits one SRT file at a time and validates dialogue assets.
- `Cinematics`: links dialogue IDs to cinematic timelines through `dialogueCues`.
Dev-only Vite endpoints in `vite.config.ts` support local saves:
- `POST /api/save-dialogues`
- `POST /api/save-srt`
- `GET /api/validate-dialogues`
- `POST /api/save-cinematics`
These endpoints are local development helpers. They are not production APIs.
## Validation
`GET /api/validate-dialogues` validates:
- manifest shape
- referenced dialogue audio files
- French SRT files
- referenced subtitle cue indexes
- optional English SRT files as warnings
Run validation after adding or renaming dialogue audio, changing cue indexes, or editing SRT files.
## Known Limitations
- There is no production persistence for audio manifests or SRT files.
- Dialogue branching is not implemented.
- Dialogue interruption and priority rules are minimal; playback is queue-based.
- SRT editing is text-based and does not yet provide waveform editing.
- Music currently supports one active looped track at a time.
+53 -4
View File
@@ -52,7 +52,7 @@ src/
## Responsibilities
`src/pages/editor/page.tsx` is the route-level composition component. It owns route-specific state such as selected object, hovered object, transform mode, and player-mode toggle.
`src/pages/editor/page.tsx` is the route-level composition component. It owns route-specific state such as selected object, hovered object, transform mode, selection lock, player-mode toggle, cinematic preview requests, and editor scene loading state.
`src/hooks/editor/useEditorSceneData.ts` loads the default map data and handles folder uploads.
@@ -62,7 +62,7 @@ src/
`src/components/editor/scene/EditorMap.tsx` renders map nodes, fallback cubes, selection highlighting, and transform controls.
`src/components/editor/EditorControls.tsx` renders the HTML control panel outside the canvas.
`src/components/editor/EditorControls.tsx` renders the HTML control panel outside the canvas. The panel is organized into top-level `details` groups: `Editor`, `Cinematics`, `Dialogues`, and `SRT`.
`src/components/editor/EditorDialogueManifestPanel.tsx` renders the dialogue manifest editor. It loads `dialogues.json`, edits dialogue entries, previews selected dialogue playback, creates missing French SRT cues, and saves the manifest through a dev-server endpoint.
@@ -122,13 +122,17 @@ If `model.glb` and `model.gltf` are both missing, the editor renders a fallback
2. `useEditorSceneData` calls `loadMapSceneData()`.
3. `loadMapSceneData()` loads `/map.json` and available model URLs.
4. If `/map.json` is missing, the page displays a folder-upload flow.
5. `EditorScene` renders the grid, lights, camera controls, and map nodes.
6. `EditorControls` exposes transform mode, history actions, export, save, and selection info.
5. `EditorSceneLoadingTracker` uses drei `useProgress()` to update the fullscreen editor loading overlay while models load.
6. `EditorScene` renders the grid, lights, camera controls, and map nodes inside `Suspense`.
7. `EditorControls` exposes transform mode, history actions, export, save, JSON preview, selection lock, and the cinematic/dialogue/SRT editors.
## Controls
- Click: select a node.
- `Esc`: clear selection.
- Click empty space: clear selection.
- Selection lock button: prevent object clicks, empty-space clicks, and `Esc` from changing the current selection.
- Selection clear button: intentionally clear the current selection even when the lock is active.
- `T`: translate mode.
- `R`: rotate mode.
- `S`: scale mode.
@@ -147,6 +151,51 @@ The editor supports two output paths:
The dev-only `/api/save-map` endpoint is implemented by the Vite plugin in `vite.config.ts`. It writes to `public/map.json` and enforces a maximum payload size.
## Editor Loading Overlay
The editor uses `SceneLoadingOverlay` like the runtime scene. `EditorSceneLoadingTracker` lives in `src/pages/editor/page.tsx` and reads drei `useProgress()` inside the canvas.
The route tracks two loading phases:
- map JSON loading through `useEditorSceneData()`
- model loading through `useProgress()`
The overlay is rendered outside the canvas so it remains visible while the R3F scene mounts. The scene itself is wrapped in `Suspense` with a `null` fallback; the visual feedback is handled by the overlay instead of by the canvas fallback.
## Panel Groups
`EditorControls` uses the local `EditorPanelGroup` helper to keep the side panel navigable as tools grow.
Current group order:
1. `Editor`
2. `Cinematics`
3. `Dialogues`
4. `SRT`
Inside the `Editor` group, the section order is:
1. `Shortcuts`
2. `Transform`
3. `Selection`
4. `View`
5. `JSON`
6. `File`
The `Shortcuts` group is nested and closed by default to reduce visual noise.
## Selection Lock
Selection lock is owned by `EditorPage` through `isSelectionLocked`.
The state is passed to:
- `EditorControls`, to render the lock/unlock button
- `EditorScene`, to block `Esc` deselection when locked
- `EditorMap`, to block object selection and empty-space deselection when locked
The clear button calls `onClearSelection` directly from `EditorControls`. This is intentionally separate from scene click behavior so the user always has an explicit way to clear the selection.
## Dialogue SRT Editing
Dialogue subtitle editing is part of the `/editor` side panel.
+14 -5
View File
@@ -4,9 +4,9 @@ This document describes the hand tracking system that exists in the current code
## Purpose
Hand tracking is a debug-stage interaction system used to test direct 3D object manipulation with a webcam. It allows a user to close their fist to grab a nearby object and move it in 3D space without relying on the center crosshair.
Hand tracking started as a debug-stage interaction system used to test direct 3D object manipulation with a webcam. It allows a user to close their fist to grab a nearby object and move it in 3D space without relying on the center crosshair.
The feature is scoped to the debug physics scene rather than production gameplay input.
It is now also available to the production repair flow when a mission reaches a hand-driven step.
## Runtime Flow
@@ -16,13 +16,13 @@ The feature is scoped to the debug physics scene rather than production gameplay
4. The backend returns hand data including landmarks, handedness, score, center point, and `isFist`.
5. React stores the latest snapshot in the hand tracking provider.
6. `GrabbableObject` reads that snapshot each frame and uses fist state plus raycasting to grab objects.
7. `HandTrackingGlove` reads the same snapshot and places the rigged `gant_l` and `gant_r` models on the detected hands in the debug physics scene.
7. `HandTrackingGlove` reads the same snapshot and places the rigged `gant_l` and `gant_r` models on the detected hands when hand tracking is active.
## Activation Rules
Hand tracking is intentionally gated so the webcam and backend are not used all the time.
The current activation conditions are:
The debug activation conditions are:
- debug mode is active with `?debug`
- scene mode is `physics`
@@ -30,6 +30,15 @@ The current activation conditions are:
This keeps hand tracking active while the player is inside an interaction zone, even if the camera is not aimed directly at the object.
The production repair activation conditions are:
- active `mainState` is `bike`, `pylone`, or `ferme`
- the active mission step is `inspected`, `repairing`, `reassembling`, or `done`
This keeps the webcam off during `waiting`, `fragmented`, and `scanning`, then enables hand input only when the repair flow is expected to use hands.
In the current production repair flow, `inspected` uses a two-fists hold gesture to advance to `fragmented`. The hold must last one second and is independent from local object interaction distance once the mission is in the correct state. Keyboard input for the same transition is handled separately by the repair case trigger, so pressing `E` requires the case to be focused through the shared interaction system.
## Backend
The backend lives in `backend/` and exposes:
@@ -121,7 +130,7 @@ The glove models are intentionally smaller than the raw SVG overlay so they do n
## Known Limitations
- The feature is debug-only and focused on the physics test scene.
- Production usage is currently limited to repair mission steps that explicitly need hands.
- MediaPipe depth is relative and can be noisy.
- The virtual hit zone is an approximation based on multiple raycasts, not a real 3D collider.
- There is no smoothing layer for hand position or depth yet.
+25 -1
View File
@@ -75,6 +75,7 @@ The mission steps currently use this sequence:
"fragmented" |
"scanning" |
"repairing" |
"reassembling" |
"done";
```
@@ -114,10 +115,32 @@ setMainState("bike");
Direct setters are useful for debug panels, but production gameplay should prefer business actions such as `advanceGameState`, `completeBike`, or `completePylone`.
Mission gameplay that can target `bike`, `pylone`, or `ferme` should prefer the generic mission actions:
```ts
const setMissionStep = useGameStore((state) => state.setMissionStep);
const completeMission = useGameStore((state) => state.completeMission);
setMissionStep("bike", "inspected");
completeMission("bike");
```
This keeps reusable gameplay components such as repair flows from duplicating mission-specific branches like `setBikeState`, `setPyloneState`, and `setFermeState`.
## World Integration
`src/world/GameStageContent.tsx` subscribes to `mainState` and mounts stage-specific content.
For repair missions, it mounts the reusable `RepairGame` component with a mission id:
```tsx
<RepairGame mission="bike" position={[8, 0, -6]} />
```
`RepairGame` reads the active mission step from the store and writes transitions through generic actions such as `setMissionStep` and `completeMission`. Shared repair ids, mission steps, and runtime guards live in `src/types/gameplay/repairMission.ts` so static mission config does not depend on the Zustand store. The production repair flow currently supports `waiting -> inspected -> fragmented -> scanning -> repairing -> reassembling -> done -> next mission` state transitions.
Mission-specific behavior stays in `src/data/gameplay/repairMissions.ts`: each mission can define its broken nodes, placeholder targets, scan duration, and reassembly duration without adding mission branches to `RepairGame`.
That means the scene can progressively move toward this pattern:
```tsx
@@ -147,6 +170,7 @@ Current overlays:
- `GameStateDebugPanel`: compact debug UI for viewing and switching main/sub states, stepping backward or forward, and resetting the store
- `Crosshair`: player aiming helper
- `InteractPrompt`: interaction prompt
- `RepairMovementLockIndicator`: player-facing indicator shown while repair steps temporarily disable movement
`src/pages/page.tsx` should stay thin and mount only the canvas and `GameUI`.
@@ -161,4 +185,4 @@ Current overlays:
## Next Steps
The next natural step is to replace the temporary stage anchors in `GameStageContent` with real stage components, for example `IntroContent`, `BikeContent`, `PyloneContent`, `FermeContent`, and `OutroContent`.
Move repair validation into mission data once each mission has distinct broken module nodes, replacement assets, and completion events.