Feat/polish-mission2 #14

Merged
math-pixel merged 5 commits from feat/polish-mission-2 into develop 2026-06-03 00:03:55 +00:00
14 changed files with 642 additions and 133 deletions
Showing only changes of commit 1325b7b2af - Show all commits
+45 -8
View File
@@ -20,9 +20,11 @@ Both sources funnel into the same `HandTrackingContext` so all consumers see one
1. The active source captures or receives landmarks.
2. The hook applies an EMA smoothing pass on the landmarks before publishing the snapshot.
3. `HandTrackingProvider` exposes that snapshot through React context.
4. `GrabbableObject` reads the snapshot each frame and uses the fist state plus raycasting to grab objects.
5. `HandTrackingGlove` reads the same snapshot and places a rigged glove on each detected hand.
6. `HandTrackingVisualizer` paints an SVG wireframe overlay on top of the canvas.
4. `GrabbableObject` reads the snapshot each frame and uses `hand.isFist` plus raycasting to grab objects.
5. `HandTrackingVisualizer` paints the SVG hand silhouette overlay on top of the canvas — the primary visualization.
6. `HandTrackingGlove` (opt-in, see UI And Debug) places a rigged 3D glove on each detected hand when enabled via the debug toggle.
All consumers — fist detection, grab raycasting, SVG silhouette, optional 3D glove — read the **same** landmarks from the snapshot. None of them depend on the others.
## Activation Rules
@@ -108,6 +110,17 @@ interface HandTrackingHand {
`x` and `y` are normalized camera coordinates. `z` is a relative depth value from MediaPipe, not an absolute world-space distance.
## Fist Detection
`isFist` is computed in `src/lib/handTracking/browserHandTracking.ts` (`isFist()` function) from landmarks alone — no model, no glove. The check is:
1. Palm center = mean of landmarks `[0, 5, 9, 13, 17]` (wrist + 4 MCPs).
2. Palm size = distance from wrist (landmark 0) to middle MCP (landmark 9).
3. For each of the four fingertip landmarks `[8, 12, 16, 20]`, check whether its distance to the palm center is less than `1.05 × palmSize`.
4. `isFist === true` iff all four fingertips pass the check.
The flag is attached to each hand on the snapshot at the publish step (`isFist: isFist(normalizedLandmarks)`) and read directly by `GrabbableObject.tsx` — the SVG visualizer and the 3D glove never participate in the gesture decision.
## Grab Targeting
The hand grab logic lives in `src/components/three/interaction/GrabbableObject.tsx`.
@@ -142,18 +155,40 @@ This is less expressive than true depth-aware hand movement, but it is more stab
The current debug UI includes:
- `HandTrackingDebugPanel` inside `DebugOverlayLayout` for status, usage, loaded glove model, server state, hand count, and fist state
- `HandTrackingVisualizer` for the SVG landmark overlay
- `HandTrackingFallback` for the last-resort hand silhouette overlay
- `HandTrackingGlove` for the per-hand rigged glove models in the R3F scene
- `HandTrackingVisualizer` for the SVG hand silhouette overlay (always on when tracking is active)
- `HandTrackingFallback` for the last-resort hand silhouette overlay (legacy, see below)
- `HandTrackingGlove` for the per-hand rigged glove models in the R3F scene, opt-in via the **Show Model** toggle
- `r3f-perf` for render performance
- `lil-gui` for scene, camera, lighting, interaction, and grab controls
The SVG visualizer uses a "blueish hand" style: white connection lines between landmarks, cyan circles with a dark blue outline. The outline gets thicker when the hand is detected as a fist, so the user gets a visual confirmation of the grab gesture without having to look at the debug panel.
### SVG Visualizer
The fallback overlay (`HandTrackingFallback`) draws a simple open-hand or fist silhouette positioned on the detected wrist landmark. It only renders for a hand whose matching glove is in the `"error"` state in `useHandTrackingGloveStatus`. This guarantees the user always sees something on their hand even when the 3D glove model fails to load.
`HandTrackingVisualizer` is the primary hand visualization. It draws a light-blue hand silhouette with a crisp dark-blue outline by:
1. Filling a palm polygon (landmarks `[1, 5, 9, 13, 17]` plus two synthetic wrist corners) and five finger tubes (thick rounded `stroke` along each finger's joint chain).
2. Wrapping the whole thing in an SVG `<filter>` that uses `feMorphology` to dilate the merged alpha by 2 px and subtract the original, producing a single continuous outline around the union — no internal seams where the palm and finger tubes overlap.
3. Shrinking every landmark toward the hand centroid by `RENDER_SCALE = 0.65` so the silhouette stays compact and doesn't dominate the screen.
4. Overlaying the 21 raw landmarks and 21 bones as faint translucent lines and dots, so the user can still see the MediaPipe data feeding the silhouette.
The SVG only displays when MediaPipe is active and the debug **Show Model** toggle is off (default). When the toggle is on, the SVG hides and `HandTrackingGlove` takes over.
### Show Model Toggle
The `Hand Tracking` debug folder exposes a single visualization switch:
- `showHandTrackingModel = false` (default): SVG visualizer renders, 3D glove is not mounted at all.
- `showHandTrackingModel = true`: SVG visualizer hides, 3D glove gets mounted for the detected hand(s).
The 3D glove is treated as opt-in legacy because it had bugs (WebGL context loss, finger rig artefacts) and its hit/grab role was never load-bearing — grab has always read landmarks directly.
### Fallback Overlay (legacy)
`HandTrackingFallback` draws a simple open-hand or fist silhouette positioned on the detected wrist landmark. It renders for any hand whose glove is in the `"error"` state in `useHandTrackingGloveStatus`. Now that the glove is opt-in and rarely mounted, the fallback effectively only fires in the rare case where the user enables `showHandTrackingModel` and the glove fails to load. It is kept on disk for that edge case but is not part of the default visual path.
## Glove Models
The 3D glove is **opt-in** via the `Show Model` debug toggle (see UI And Debug). It is not mounted by default; the SVG visualizer is the primary hand UI. The information below applies only when the toggle is enabled.
`HandTrackingGlove` loads `public/models/gant_l/model.gltf` for both hands. The right hand applies `scale.x = -1` at the group level to mirror the mesh, so the thumb ends up on the correct side. Both hands therefore share the same rig and the same material.
The historical `public/models/gant_r/model.gltf` is kept as legacy but is not loaded by the frontend — its GLB embeds three skeletons (`Hand_l`, `Hand_l_pad`, `Hand_r`) plus a `galet` mesh, which made the finger rig unreliable.
@@ -172,6 +207,8 @@ They are intended for future swap-by-state usage but are **not yet rigged**. The
- Production usage is currently limited to repair mission steps that explicitly need hands.
- MediaPipe depth is relative and currently not used for stable object depth control.
- The virtual hit zone is an approximation based on multiple raycasts, not a real 3D collider.
- The 3D glove is opt-in only (see `Show Model` toggle). Default visual is the SVG silhouette.
- `HandTrackingFallback` is legacy and effectively unused unless the glove toggle is enabled and the glove fails to load.
- The right glove is a mirrored copy of `gant_l` rather than its own mesh; in the future a dedicated right-hand model would give a better visual.
- The `_pad` glove variants are not rigged yet, so swap-by-state (normal ↔ pad) is not wired in.
- Finger bone animation is an approximate landmark-to-bone mapping; it still needs calibration for per-model twist, offsets, and smoothing.
+8 -2
View File
@@ -334,7 +334,7 @@ export function Ebike({
const interactionLabel =
mainState === "ebike"
? "Lancer le repair game"
? "Lancer le Repair Game"
: movementMode === "walk"
? "Monter sur le bike"
: "Descendre du bike";
@@ -344,13 +344,19 @@ export function Ebike({
// pollute the view. The prompt comes back the moment the bike comes to
// a stop. window.ebikeDriveInputActive is published every frame by
// PlayerController based on whether a movement key is currently held.
// Also hide entirely while the breakdown sequence is active — the bike
// must read as inert and non-interactive while the panne dialogue plays
// and during the auto-dismount that follows.
const [isEbikeDriving, setIsEbikeDriving] = useState(false);
const [isEbikeBreakdown, setIsEbikeBreakdown] = useState(false);
useFrame(() => {
const driving =
movementMode === "ebike" && window.ebikeDriveInputActive === true;
if (driving !== isEbikeDriving) setIsEbikeDriving(driving);
const breakdown = window.ebikeBreakdownActive === true;
if (breakdown !== isEbikeBreakdown) setIsEbikeBreakdown(breakdown);
});
const showInteractPrompt = !isEbikeDriving;
const showInteractPrompt = !isEbikeDriving && !isEbikeBreakdown;
const handleInteract = useCallback((): void => {
if (window.ebikeBreakdownActive === true) return;
+3 -3
View File
@@ -22,8 +22,6 @@ export function SiteCard({
return "#b8b8b8";
};
const borderColor = selected ? "#a8d5a2" : "rgba(255, 255, 255, 0.55)";
const textColor = disabled ? "rgba(77, 77, 77, 0.72)" : "#4d4d4d";
return (
@@ -41,7 +39,9 @@ export function SiteCard({
height: isSituation
? "clamp(48px, 6vw, 60px)"
: "clamp(140px, 18vw, 180px)",
border: `3px solid ${borderColor}`,
border: "3px solid rgba(255, 255, 255, 0.55)",
outline: selected ? "3px solid #a8d5a2" : "none",
outlineOffset: 0,
background: getBackground(),
cursor: disabled ? "not-allowed" : "pointer",
display: "flex",
+103 -48
View File
@@ -1,59 +1,133 @@
import { useCallback, useEffect, useRef, useState } from "react";
import { useEffect, useState } from "react";
import { useGameStore } from "@/managers/stores/useGameStore";
import { useSiteStore } from "@/managers/stores/useSiteStore";
import { useSettingsStore } from "@/managers/stores/useSettingsStore";
import { SiteButton } from "@/components/site/SiteButton";
import { SITE_CONFIG } from "@/data/site/siteConfig";
import { SITE_DIALOGUE_IDS } from "@/data/site/dialogueIds";
import { loadDialogueManifest } from "@/utils/dialogues/loadDialogueManifest";
import {
loadDialogueManifest,
loadDialogueSubtitleCues,
} from "@/utils/dialogues/loadDialogueManifest";
import {
playDialogueById,
stopCurrentDialogue,
} from "@/utils/dialogues/playDialogue";
const TYPEWRITER_CHAR_DELAY_MS = 70;
// Fallback in case nothing else triggers the typewriter (audio failed to
// load, no subtitles, "ended" never fires). Long enough not to fire
// before the narration on a slow load.
const AUDIO_END_FALLBACK_MS = 8000;
/**
* Screen 3: Name input
* The displayed name is forced to SITE_CONFIG.presetPlayerName — the
* field reveals one letter per keystroke until the preset name is complete.
* Screen 3: Name reveal
* The player's preset name is revealed letter-by-letter inside the input
* once the naming dialogue finishes playing. The confirm button stays
* locked until the reveal completes. No user typing — the input is
* read-only and just acts as a typewriter target.
*/
export function SiteNamingScreen(): React.JSX.Element {
const setStep = useSiteStore((state) => state.setStep);
const setPlayerName = useGameStore((state) => state.setPlayerName);
const [charIndex, setCharIndex] = useState(0);
const inputRef = useRef<HTMLInputElement>(null);
const [revealedChars, setRevealedChars] = useState(0);
const [typewriterStarted, setTypewriterStarted] = useState(false);
const presetPlayerName = SITE_CONFIG.presetPlayerName;
const displayValue = presetPlayerName.slice(0, charIndex);
const isComplete = charIndex >= presetPlayerName.length;
const displayValue = presetPlayerName.slice(0, revealedChars);
const isComplete = revealedChars >= presetPlayerName.length;
// Play the dialogue, then trigger the typewriter so it FINISHES at the
// same moment the narration ends. We compute that moment from the SRT
// cues: the last cue's endTime is where the narrator stops speaking,
// so we start typing `typewriterDuration` before that.
useEffect(() => {
let cancelled = false;
let audioElement: HTMLAudioElement | null = null;
let onTimeUpdate: (() => void) | null = null;
let fallbackTimer: ReturnType<typeof setTimeout> | null = null;
const start = (): void => {
if (cancelled) return;
setTypewriterStarted(true);
};
const typewriterDurationSec =
(TYPEWRITER_CHAR_DELAY_MS * presetPlayerName.length) / 1000;
void (async () => {
const manifest = await loadDialogueManifest();
if (cancelled || !manifest) return;
await playDialogueById(manifest, SITE_DIALOGUE_IDS.naming);
if (cancelled) return;
if (!manifest) {
start();
return;
}
// Resolve the dialogue + its SRT cues for the active subtitle language.
const dialogue = manifest.dialogues.find(
(item) => item.id === SITE_DIALOGUE_IDS.naming,
);
const language = useSettingsStore.getState().subtitleLanguage;
const subtitleData = dialogue
? await loadDialogueSubtitleCues(manifest, dialogue, language)
: null;
if (cancelled) return;
audioElement = await playDialogueById(manifest, SITE_DIALOGUE_IDS.naming);
if (cancelled) return;
if (!audioElement) {
start();
return;
}
const lastCue = subtitleData?.cues[subtitleData.cues.length - 1];
if (lastCue) {
// Trigger so the typewriter ends at the narration's end.
const audio = audioElement;
const triggerAt = Math.max(0, lastCue.endTime - typewriterDurationSec);
onTimeUpdate = (): void => {
if (audio.currentTime >= triggerAt) {
audio.removeEventListener("timeupdate", onTimeUpdate!);
start();
}
};
audio.addEventListener("timeupdate", onTimeUpdate);
} else {
// No SRT data — fall back to the audio "ended" event.
audioElement.addEventListener("ended", start, { once: true });
}
fallbackTimer = setTimeout(start, AUDIO_END_FALLBACK_MS);
})();
return () => {
cancelled = true;
if (fallbackTimer !== null) clearTimeout(fallbackTimer);
if (audioElement) {
if (onTimeUpdate) {
audioElement.removeEventListener("timeupdate", onTimeUpdate);
}
audioElement.removeEventListener("ended", start);
}
stopCurrentDialogue();
};
}, []);
}, [presetPlayerName.length]);
// Reveal the preset name one character at a time once the typewriter
// has been triggered.
useEffect(() => {
inputRef.current?.focus();
}, []);
const handleNameChange = useCallback(
(event: React.ChangeEvent<HTMLInputElement>): void => {
const nextLength = Math.min(
event.target.value.length,
presetPlayerName.length,
);
setCharIndex(nextLength);
},
[presetPlayerName.length],
);
if (!typewriterStarted) return;
const interval = setInterval(() => {
setRevealedChars((current) => {
if (current >= presetPlayerName.length) {
clearInterval(interval);
return current;
}
return current + 1;
});
}, TYPEWRITER_CHAR_DELAY_MS);
return () => clearInterval(interval);
}, [typewriterStarted, presetPlayerName.length]);
const handleConfirm = (): void => {
if (isComplete) {
@@ -98,17 +172,16 @@ export function SiteNamingScreen(): React.JSX.Element {
margin: 0,
}}
>
Quel est votre prénom ?
Je suis
</h2>
<input
ref={inputRef}
type="text"
value={displayValue}
onChange={handleNameChange}
placeholder="Écrivez votre prénom ici"
readOnly
tabIndex={-1}
aria-labelledby="player-name-label"
aria-describedby="player-name-hint"
aria-live="polite"
autoComplete="off"
style={{
display: "flex",
@@ -122,30 +195,12 @@ export function SiteNamingScreen(): React.JSX.Element {
background: "#D9D9D9",
outline: "none",
color: "#333",
caretColor: "#333",
fontFamily: "Inter, system-ui, sans-serif",
fontSize: "clamp(16px, 2.5vw, 20px)",
textAlign: "left",
boxSizing: "border-box",
}}
/>
<span
id="player-name-hint"
style={{
position: "absolute",
width: 1,
height: 1,
padding: 0,
margin: -1,
overflow: "hidden",
clip: "rect(0, 0, 0, 0)",
whiteSpace: "nowrap",
border: 0,
}}
>
Votre personnage s&apos;appelle {presetPlayerName}. Tapez{" "}
{presetPlayerName.length} caractères pour révéler son nom.
</span>
</div>
<SiteButton
+4
View File
@@ -7,6 +7,8 @@ import { InteractPrompt } from "@/components/ui/InteractPrompt";
import { OutroVideoOverlay } from "@/components/ui/OutroVideoOverlay";
import { Subtitles } from "@/components/ui/Subtitles";
import { TalkieDialogueOverlay } from "@/components/ui/TalkieDialogueOverlay";
import { HandTrackingTutorial } from "@/components/ui/tutorial/HandTrackingTutorial";
import { MovementTutorial } from "@/components/ui/tutorial/MovementTutorial";
export function GameUI(): React.JSX.Element {
return (
@@ -16,6 +18,8 @@ export function GameUI(): React.JSX.Element {
<InteractPrompt />
<HandTrackingVisualizer />
<HandTrackingFallback />
<MovementTutorial />
<HandTrackingTutorial />
<Subtitles />
<TalkieDialogueOverlay />
<GameSettingsMenu />
+59 -18
View File
@@ -4,29 +4,70 @@ import {
type HandTrackingGloveHandedness,
} from "@/hooks/handTracking/useHandTrackingGloveStatus";
// Simple schematic silhouettes used as a last-resort fallback when the
// rigged glove model has failed to load. Both icons share the same
// 48x48 viewBox and the same stroke/fill rules from the .css.
// Hand silhouettes used as a last-resort fallback when the rigged glove
// model has failed to load. Both icons share a 100x120 viewBox so finger
// lengths and the thumb angle stay anatomically readable.
const OpenHandShape = (): React.JSX.Element => (
<>
<ellipse cx="9" cy="30" rx="3" ry="6" transform="rotate(-25 9 30)" />
<rect x="14" y="8" width="4" height="22" rx="2" />
<rect x="20" y="4" width="4" height="26" rx="2" />
<rect x="26" y="6" width="4" height="24" rx="2" />
<rect x="32" y="10" width="4" height="20" rx="2" />
<rect x="10" y="26" width="28" height="18" rx="6" />
</>
<path
d="M 28 116
Q 22 100 22 80
Q 22 65 28 58
Q 22 52 14 46
Q 6 40 8 28
Q 12 18 22 20
Q 30 24 30 36
Q 32 46 36 50
Q 36 38 36 28
Q 36 18 42 18
Q 48 18 48 28
Q 48 40 50 50
Q 50 32 50 14
Q 50 6 56 6
Q 62 6 62 14
Q 62 32 62 50
Q 64 38 64 20
Q 64 12 70 12
Q 76 12 76 20
Q 76 38 78 50
Q 78 40 78 32
Q 78 24 84 24
Q 90 24 90 32
Q 90 44 92 56
Q 96 80 92 100
Q 86 116 82 116
Z"
/>
);
const FistShape = (): React.JSX.Element => (
<>
<ellipse cx="8" cy="26" rx="3" ry="5" />
<rect x="10" y="14" width="28" height="30" rx="10" />
<circle cx="15" cy="14" r="3" />
<circle cx="21" cy="13" r="3" />
<circle cx="27" cy="13" r="3" />
<circle cx="33" cy="14" r="3" />
<path
d="M 18 70
Q 14 50 24 38
Q 28 30 36 34
Q 40 26 48 30
Q 54 22 60 28
Q 68 24 74 32
Q 84 32 88 46
Q 92 64 88 82
Q 82 104 64 112
Q 42 116 26 108
Q 14 96 18 70
Z"
/>
<path
d="M 18 70
Q 6 66 8 80
Q 8 94 18 96
Q 28 94 26 84
Q 22 76 18 70
Z"
/>
<path d="M 32 38 Q 30 50 34 60" fill="none" strokeLinecap="round" />
<path d="M 46 32 Q 44 46 48 58" fill="none" strokeLinecap="round" />
<path d="M 60 32 Q 58 46 62 58" fill="none" strokeLinecap="round" />
<path d="M 74 36 Q 72 50 76 60" fill="none" strokeLinecap="round" />
</>
);
@@ -66,7 +107,7 @@ export function HandTrackingFallback(): React.JSX.Element | null {
<svg
key={`${handedness}-${index}`}
className="hand-tracking-fallback__icon"
viewBox="0 0 48 48"
viewBox="0 0 100 120"
style={{
left: `${leftPercent}%`,
top: `${topPercent}%`,
+171 -46
View File
@@ -1,8 +1,16 @@
import { useHandTrackingSnapshot } from "@/hooks/handTracking/useHandTrackingSnapshot";
import { useHandTrackingGloveStatus } from "@/hooks/handTracking/useHandTrackingGloveStatus";
import { useDebugStore } from "@/hooks/debug/useDebugStore";
const HAND_CONNECTIONS: Array<[number, number]> = [
// MediaPipe indexes the 21 hand landmarks predictably:
// 0 wrist, 1-4 thumb (base→tip), 5-8 index, 9-12 middle, 13-16 ring, 17-20 pinky.
const FINGER_LANDMARKS: Array<readonly number[]> = [
[1, 2, 3, 4],
[5, 6, 7, 8],
[9, 10, 11, 12],
[13, 14, 15, 16],
[17, 18, 19, 20],
];
const SKELETON_BONES: Array<[number, number]> = [
[0, 1],
[1, 2],
[2, 3],
@@ -26,70 +34,187 @@ const HAND_CONNECTIONS: Array<[number, number]> = [
[0, 17],
];
const LANDMARK_FILL = "#67e8f9"; // cyan-300, opaque interior
const LANDMARK_STROKE = "#0c4a6e"; // sky-900, dark blue outline
const LANDMARK_STROKE_FIST = "#1e3a8a"; // blue-900, thicker accent when fist
const CONNECTION_STROKE = "#ffffff"; // white bones
const INDEX_TIP_LANDMARK = 8;
const HAND_FILL = "#bfdbfe"; // blue-200, light interior
const HAND_OUTLINE_COLOR = "#1e3a8a"; // blue-900, crisp dark outline
const HAND_OUTLINE_RADIUS = 2; // px
// Shrink the rendered hand around its centroid. Grab/physics keep using raw
// landmarks elsewhere, so the silhouette is just visually smaller.
const RENDER_SCALE = 0.65;
const FINGER_THICKNESS_FACTOR = 0.08; // fraction of (scaled) hand length
const WRIST_HALF_WIDTH = 0.28;
const SKELETON_STROKE = "rgba(30, 58, 138, 0.22)";
const SKELETON_DOT_FILL = "rgba(30, 58, 138, 0.35)";
const FILTER_ID = "hand-tracking-outline";
export function HandTrackingVisualizer(): React.JSX.Element | null {
const { hands, status } = useHandTrackingSnapshot();
const showHandTrackingSvg = useDebugStore((debug) =>
debug.getShowHandTrackingSvg(),
);
const gloves = useHandTrackingGloveStatus((state) => state.gloves);
const hasLoadedGlove = Object.values(gloves).some(
(gloveStatus) => gloveStatus === "loaded",
const showHandTrackingModel = useDebugStore((debug) =>
debug.getShowHandTrackingModel(),
);
if (
status === "idle" ||
hands.length === 0 ||
(hasLoadedGlove && !showHandTrackingSvg)
) {
if (status === "idle" || hands.length === 0 || showHandTrackingModel) {
return null;
}
const viewportWidth = window.innerWidth;
const viewportHeight = window.innerHeight;
return (
<svg className="hand-tracking-visualizer" aria-hidden="true">
<defs>
{/* Dilate the merged alpha of all child shapes by HAND_OUTLINE_RADIUS
and subtract the original to get a 1-ring outline. Lets the palm
polygon and the five finger tubes share a single crisp outline
with no internal seams where they overlap. */}
<filter id={FILTER_ID} x="-10%" y="-10%" width="120%" height="120%">
<feMorphology
operator="dilate"
radius={HAND_OUTLINE_RADIUS}
in="SourceAlpha"
result="dilated"
/>
<feComposite
operator="out"
in="dilated"
in2="SourceAlpha"
result="ringAlpha"
/>
<feFlood floodColor={HAND_OUTLINE_COLOR} result="ringColor" />
<feComposite
operator="in"
in="ringColor"
in2="ringAlpha"
result="coloredRing"
/>
<feMerge>
<feMergeNode in="SourceGraphic" />
<feMergeNode in="coloredRing" />
</feMerge>
</filter>
</defs>
{hands.map((hand, handIndex) => {
const landmarks = hand.landmarks;
if (landmarks.length === 0) return null;
if (landmarks.length < 21) return null;
const landmarkStroke = hand.isFist
? LANDMARK_STROKE_FIST
: LANDMARK_STROKE;
// Centroid of all 21 landmarks in pixel space (mirrored x).
let cx = 0;
let cy = 0;
for (const lm of landmarks) {
cx += (1 - lm.x) * viewportWidth;
cy += lm.y * viewportHeight;
}
cx /= landmarks.length;
cy /= landmarks.length;
// Render coordinates: shrink each landmark toward the centroid.
const px = (i: number): number => {
const lm = landmarks[i];
return lm
? cx + ((1 - lm.x) * viewportWidth - cx) * RENDER_SCALE
: cx;
};
const py = (i: number): number => {
const lm = landmarks[i];
return lm ? cy + (lm.y * viewportHeight - cy) * RENDER_SCALE : cy;
};
const handLengthPx = Math.hypot(px(12) - px(0), py(12) - py(0));
const fingerThickness = Math.max(
6,
handLengthPx * FINGER_THICKNESS_FACTOR,
);
const halfFingerThickness = fingerThickness / 2;
const dotRadius = Math.max(1.2, fingerThickness * 0.1);
// Perpendicular to the palm centerline (wrist → middle MCP), used to
// place two synthetic wrist corners on either side of landmark 0.
const cdx = px(9) - px(0);
const cdy = py(9) - py(0);
const clen = Math.hypot(cdx, cdy) || 1;
const perpX = -cdy / clen;
const perpY = cdx / clen;
const thumbSide =
(px(1) - px(0)) * perpX + (py(1) - py(0)) * perpY >= 0 ? 1 : -1;
const wristHalfWidth = handLengthPx * WRIST_HALF_WIDTH;
const wristThumbX = px(0) + perpX * wristHalfWidth * thumbSide;
const wristThumbY = py(0) + perpY * wristHalfWidth * thumbSide;
const wristPinkyX = px(0) - perpX * wristHalfWidth * thumbSide;
const wristPinkyY = py(0) - perpY * wristHalfWidth * thumbSide;
// Palm outline: straight L between adjacent MCPs along the top (no
// inter-finger dip — the morphology dilation rounds the MCP corners),
// rounded heel via two Q curves bowing out to the synthetic wrist
// corners.
const palmD = [
`M ${px(1)} ${py(1)}`,
`L ${px(5)} ${py(5)}`,
`L ${px(9)} ${py(9)}`,
`L ${px(13)} ${py(13)}`,
`L ${px(17)} ${py(17)}`,
`Q ${wristPinkyX} ${wristPinkyY}, ${px(0)} ${py(0)}`,
`Q ${wristThumbX} ${wristThumbY}, ${px(1)} ${py(1)}`,
"Z",
].join(" ");
// Each finger path starts halfFingerThickness inside the palm (toward
// the next joint), so the rounded base cap sits hidden inside the palm
// fill instead of bulging below the MCP.
const fingerPathD = (joints: readonly number[]): string => {
const baseIdx = joints[0];
const nextIdx = joints[1];
if (baseIdx === undefined || nextIdx === undefined) return "";
const baseX = px(baseIdx);
const baseY = py(baseIdx);
const nextX = px(nextIdx);
const nextY = py(nextIdx);
const dx = nextX - baseX;
const dy = nextY - baseY;
const dlen = Math.hypot(dx, dy) || 1;
const sx = baseX + (dx / dlen) * halfFingerThickness;
const sy = baseY + (dy / dlen) * halfFingerThickness;
return joints
.map((idx, k) =>
k === 0 ? `M ${sx} ${sy}` : `L ${px(idx)} ${py(idx)}`,
)
.join(" ");
};
return (
<g key={`${hand.handedness}-${handIndex}`}>
{HAND_CONNECTIONS.map(([from, to]) => {
const fromPoint = landmarks[from];
const toPoint = landmarks[to];
if (!fromPoint || !toPoint) return null;
return (
<line
key={`${from}-${to}`}
x1={`${(1 - fromPoint.x) * 100}%`}
y1={`${fromPoint.y * 100}%`}
x2={`${(1 - toPoint.x) * 100}%`}
y2={`${toPoint.y * 100}%`}
stroke={CONNECTION_STROKE}
strokeWidth="2.5"
<g filter={`url(#${FILTER_ID})`}>
<path d={palmD} fill={HAND_FILL} />
{FINGER_LANDMARKS.map((joints, fingerIndex) => (
<path
key={fingerIndex}
d={fingerPathD(joints)}
fill="none"
stroke={HAND_FILL}
strokeWidth={fingerThickness}
strokeLinecap="round"
strokeLinejoin="round"
/>
);
})}
))}
</g>
{landmarks.map((landmark, landmarkIndex) => (
{SKELETON_BONES.map(([from, to]) => (
<line
key={`bone-${from}-${to}`}
x1={px(from)}
y1={py(from)}
x2={px(to)}
y2={py(to)}
stroke={SKELETON_STROKE}
strokeWidth="1"
/>
))}
{landmarks.map((_, landmarkIndex) => (
<circle
key={landmarkIndex}
cx={`${(1 - landmark.x) * 100}%`}
cy={`${landmark.y * 100}%`}
r={landmarkIndex === INDEX_TIP_LANDMARK ? 6 : 4}
fill={LANDMARK_FILL}
stroke={landmarkStroke}
strokeWidth={hand.isFist ? 2.5 : 2}
key={`dot-${landmarkIndex}`}
cx={px(landmarkIndex)}
cy={py(landmarkIndex)}
r={dotRadius}
fill={SKELETON_DOT_FILL}
/>
))}
</g>
@@ -0,0 +1,59 @@
import { useEffect, useState } from "react";
import { Hand } from "lucide-react";
import { useGameStore } from "@/managers/stores/useGameStore";
import { useHandTrackingSnapshot } from "@/hooks/handTracking/useHandTrackingSnapshot";
import type { MissionStep } from "@/types/gameplay/repairMission";
import { TutorialOverlay } from "@/components/ui/tutorial/TutorialOverlay";
// Repair steps where the hand-tracking tutorial is allowed to display. Covers
// the no-hand-tracking phase (fragmented, scanning) and the first hand-driven
// step (inspected) — beyond that the player has presumably learned.
const HAND_TUTORIAL_STEPS: ReadonlySet<MissionStep> = new Set([
"fragmented",
"scanning",
"inspected",
]);
/**
* First-time hand-tracking tutorial. Visible during the early ebike repair
* steps until MediaPipe actually detects a hand on screen. Once dismissed it
* stays dismissed for the session.
*/
export function HandTrackingTutorial(): React.JSX.Element | null {
const mainState = useGameStore((state) => state.mainState);
const ebikeStep = useGameStore((state) => state.ebike.currentStep);
const { hands, status } = useHandTrackingSnapshot();
const [dismissed, setDismissed] = useState(false);
const isInShowWindow =
mainState === "ebike" && HAND_TUTORIAL_STEPS.has(ebikeStep);
const handsDetected = status !== "idle" && hands.length > 0;
useEffect(() => {
if (handsDetected && !dismissed) {
// Sync the persistent dismissal flag with an external signal (the
// hand-tracking snapshot). Same shape as the resync pattern used
// elsewhere in the repo (e.g. PylonDownedPylon).
// eslint-disable-next-line react-hooks/set-state-in-effect
setDismissed(true);
}
}, [handsDetected, dismissed]);
if (!isInShowWindow || dismissed) return null;
return (
<TutorialOverlay
icon={
<div className="tutorial-overlay__hands">
<Hand size={96} strokeWidth={1.5} />
<Hand
size={96}
strokeWidth={1.5}
style={{ transform: "scaleX(-1)" }}
/>
</div>
}
text="Placez vos mains devant la caméra pour attraper les pièces. Sinon, utilisez la souris."
/>
);
}
@@ -0,0 +1,57 @@
import { useEffect, useState } from "react";
import { useGameStore } from "@/managers/stores/useGameStore";
import type { GameStep } from "@/types/game";
import { TutorialOverlay } from "@/components/ui/tutorial/TutorialOverlay";
const MOVEMENT_KEYS = new Set(["z", "q", "s", "d"]);
// Intro steps where the movement tutorial is allowed to display. From the
// reveal fade through the free-walk window before the ebike mount.
const MOVEMENT_TUTORIAL_STEPS: ReadonlySet<GameStep> = new Set([
"reveal",
"await-ebike-mount",
]);
function KeyCap({ label }: { label: string }): React.JSX.Element {
return <span className="tutorial-overlay__keycap">{label}</span>;
}
/**
* First-time movement tutorial. Visible during the intro reveal and the
* walk-around step before the ebike mount, until the player presses any
* of Z, Q, S, D. Once dismissed it stays dismissed for the session.
*/
export function MovementTutorial(): React.JSX.Element | null {
const introStep = useGameStore((state) => state.intro.currentStep);
const [dismissed, setDismissed] = useState(false);
const isInShowWindow = MOVEMENT_TUTORIAL_STEPS.has(introStep);
useEffect(() => {
if (dismissed) return;
function onKeyDown(event: KeyboardEvent): void {
if (MOVEMENT_KEYS.has(event.key.toLowerCase())) {
setDismissed(true);
}
}
window.addEventListener("keydown", onKeyDown);
return () => window.removeEventListener("keydown", onKeyDown);
}, [dismissed]);
if (!isInShowWindow || dismissed) return null;
return (
<TutorialOverlay
icon={
<div className="tutorial-overlay__keyboard">
<span aria-hidden="true" />
<KeyCap label="Z" />
<span aria-hidden="true" />
<KeyCap label="Q" />
<KeyCap label="S" />
<KeyCap label="D" />
</div>
}
text="Utilisez le clavier et la souris pour vous déplacer."
/>
);
}
@@ -0,0 +1,23 @@
interface TutorialOverlayProps {
icon: React.ReactNode;
text: string;
}
/**
* Full-screen instructional overlay shown during onboarding moments
* (movement intro, hand-tracking intro, ...). Pure presentation: parent
* decides when to mount it and when to unmount it.
*/
export function TutorialOverlay({
icon,
text,
}: TutorialOverlayProps): React.JSX.Element {
return (
<div className="tutorial-overlay" aria-live="polite">
<div className="tutorial-overlay__panel">
<div className="tutorial-overlay__icon">{icon}</div>
<p className="tutorial-overlay__text">{text}</p>
</div>
</div>
);
}
+2 -2
View File
@@ -15,11 +15,11 @@ export const EBIKE_DROP_PLAYER_TRANSFORM: CameraTransform = {
rotation: [0, 0, 0],
};
export const EBIKE_WORLD_POSITION: Vector3Tuple = [65, 0.8, 72];
export const EBIKE_WORLD_POSITION: Vector3Tuple = [68, 0.8, 65];
export const EBIKE_WORLD_ROTATION_Y = -2.5;
export const EBIKE_WORLD_SCALE = 0.35;
export const EBIKE_INTRO_BREAKDOWN_DISTANCE = 15;
export const EBIKE_INTRO_BREAKDOWN_DISTANCE = 50;
export const EBIKE_BREAKDOWN_DIALOGUE_DELAY_MS = 250;
export const EBIKE_ACCELERATION_DURATION_MS = 2000;
+72 -3
View File
@@ -1799,7 +1799,8 @@ canvas {
width: 100vw;
height: 100vh;
pointer-events: none;
filter: drop-shadow(0 0 8px rgba(56, 189, 248, 0.55));
opacity: 0.8;
filter: drop-shadow(0 0 4px rgba(96, 165, 250, 0.3));
}
.hand-tracking-fallback {
@@ -1811,14 +1812,82 @@ canvas {
pointer-events: none;
}
.tutorial-overlay {
position: fixed;
inset: 0;
z-index: 14;
display: flex;
align-items: center;
justify-content: center;
background: rgba(96, 165, 250, 0.55);
pointer-events: none;
}
.tutorial-overlay__panel {
display: flex;
flex-direction: column;
align-items: center;
gap: 36px;
padding: 56px 72px;
max-width: 640px;
background: transparent;
border: 2px solid #1e3a8a;
border-radius: 24px;
color: #1e3a8a;
}
.tutorial-overlay__icon {
display: flex;
align-items: center;
justify-content: center;
}
.tutorial-overlay__text {
font-family: var(--font-body);
font-size: 1.1rem;
font-weight: 500;
line-height: 1.45;
text-align: center;
margin: 0;
}
.tutorial-overlay__keyboard {
display: grid;
grid-template-columns: repeat(3, 64px);
gap: 8px;
font-family: var(--font-primary);
}
.tutorial-overlay__keycap {
display: flex;
align-items: center;
justify-content: center;
width: 64px;
height: 64px;
background: #e0f2fe;
border: 2px solid #1e3a8a;
border-radius: 10px;
font-size: 1.6rem;
font-weight: 700;
color: #1e3a8a;
}
.tutorial-overlay__hands {
display: flex;
align-items: center;
gap: 32px;
color: #1e3a8a;
}
.hand-tracking-fallback__icon {
position: absolute;
width: 96px;
width: 80px;
height: 96px;
fill: #67e8f9;
stroke: #0c4a6e;
stroke-width: 2;
stroke-width: 3;
stroke-linejoin: round;
stroke-linecap: round;
filter: drop-shadow(0 0 8px rgba(56, 189, 248, 0.55));
}
+31 -2
View File
@@ -1,14 +1,43 @@
import { useEffect } from "react";
import { AudioManager } from "@/managers/AudioManager";
import { useGameStore } from "@/managers/stores/useGameStore";
import type { MissionStep } from "@/types/gameplay/repairMission";
const GAME_MUSIC_PATH = "/sounds/musique/musique-jeu.mp3";
const GAME_MUSIC_VOLUME = 0.33;
const REPAIR_MUSIC_PATH = "/sounds/musique/musique-reparation.mp3";
const MUSIC_VOLUME = 0.33;
// Steps during which the repair mini-game owns the experience.
// Triggered when any mission (ebike / pylon / farm) is in this range.
const REPAIR_MUSIC_STEPS: ReadonlySet<MissionStep> = new Set([
"inspected",
"fragmented",
"scanning",
"repairing",
"reassembling",
"done",
]);
export function GameMusic(): null {
const ebikeStep = useGameStore((state) => state.ebike.currentStep);
const pylonStep = useGameStore((state) => state.pylon.currentStep);
const farmStep = useGameStore((state) => state.farm.currentStep);
const inRepair =
REPAIR_MUSIC_STEPS.has(ebikeStep) ||
REPAIR_MUSIC_STEPS.has(pylonStep) ||
REPAIR_MUSIC_STEPS.has(farmStep);
useEffect(() => {
const audio = AudioManager.getInstance();
audio.playMusic(GAME_MUSIC_PATH, GAME_MUSIC_VOLUME);
audio.playMusic(
inRepair ? REPAIR_MUSIC_PATH : GAME_MUSIC_PATH,
MUSIC_VOLUME,
);
}, [inRepair]);
useEffect(() => {
const audio = AudioManager.getInstance();
return () => {
audio.stopMusic();
};
+5 -1
View File
@@ -363,7 +363,11 @@ export function PlayerController({
}
_wishDir.set(0, 0, 0);
if (!isEbikeBreakdown) {
// Block drive input only when still on the bike during breakdown.
// Once auto-dismounted (movementMode === "walk"), the player must
// remain free to walk around even though ebikeBreakdownActive is true.
const blockDriveInput = isEbikeMounted && isEbikeBreakdown;
if (!blockDriveInput) {
if (keys.current.forward) _wishDir.add(_forward);
if (keys.current.backward) _wishDir.sub(_forward);
if (!isEbikeMounted) {