La-Fabrik/docs/technical/interaction.md

Interaction System Technical Notes

This document explains the shared trigger, grab, focus, and hand-grab system.

Purpose

The app has several ways for the player to affect the 3D scene:

• press E on focused trigger objects
• hold the primary mouse button on grabbable objects
• close a tracked hand into a fist to grab hand-controlled objects
• release objects and optionally snap them into target positions

The implementation keeps those rules in a reusable interaction layer so gameplay features such as the repair game do not each create their own input system.

Main Files

File	Responsibility
src/managers/InteractionManager.ts	Shared interaction state and imperative actions
src/hooks/interaction/useInteraction.ts	React subscription to the manager
src/components/three/interaction/InteractableObject.tsx	Distance/raycast focus detection
src/components/three/interaction/TriggerObject.tsx	Press-to-trigger wrapper
src/components/three/interaction/GrabbableObject.tsx	Physics-backed grab and hand grab wrapper
src/components/ui/InteractPrompt.tsx	HTML prompt for focused trigger interactions
src/world/player/PlayerController.tsx	Keyboard/mouse input bridge

Architecture

The interaction system has three layers:

1. R3F objects detect focus and register handles.
2. InteractionManager stores the current interaction snapshot.
3. UI and player input read the snapshot and trigger the selected action.

This is intentionally not Zustand. Interaction focus and holding state are short-lived, frame-adjacent runtime state. A small singleton plus useSyncExternalStore is a better fit than putting high-frequency interaction details into the durable game progression store.

Interaction Snapshot

The snapshot type lives in:

src/types/interaction/interaction.ts

interface InteractionSnapshot {
  focused: InteractableHandle | null;
  nearby: boolean;
  holding: boolean;
  handHolding: boolean;
}

Meaning:

• focused: the interactable currently aimed at by the camera ray
• nearby: at least one interactable is within interaction radius
• holding: mouse/player-controller grab is active
• handHolding: hand-tracking grab is active

nearby, holding, and handHolding are also used by the hand-tracking provider to decide when webcam tracking should stay active in the debug physics scene.

Focus Detection

Focus detection lives in:

src/components/three/interaction/InteractableObject.tsx

Each frame, it:

1. finds the interactable world position from its Rapier body or group transform
2. checks distance from the camera
3. marks the handle as nearby if it is inside radius
4. raycasts from the camera forward direction
5. sets the focused handle when the ray hits the object
6. clears focus if the object is no longer nearby or no longer aimed at

This gives a simple first-person interaction model: the player must be close enough and looking at the object.

Trigger Objects

Trigger implementation:

src/components/three/interaction/TriggerObject.tsx

TriggerObject wraps children in a fixed Rapier body and exposes a trigger handle.

When triggered, it can:

• play an optional SFX through AudioManager
• call onTrigger
• spawn an optional model at an offset

Typical users:

• repair-object inspection
• repair-case open/fragment interaction
• install target
• completion target
• debug scene trigger sphere

Grabbable Objects

Grab implementation:

src/components/three/interaction/GrabbableObject.tsx

GrabbableObject wraps children in a dynamic Rapier body and exposes a grab handle.

Mouse/controller grab flow:

1. Player focuses the object.
2. Mouse down calls InteractionManager.pressInteract().
3. The object enters holding mode.
4. Each frame, velocity is pushed toward a hold target in front of the camera.
5. Mouse up calls releaseInteract().
6. The object can snap to the nearest configured target.

Important tuning values live in:

src/data/interaction/grabConfig.ts

The debug GUI exposes hold stiffness, throw boost, and hold distance.

Snap-To-Target

GrabbableObject supports:

• snapTargets
• snapRadius
• snapDuration
• onSnap

On release, the object finds the nearest target inside snapRadius. If a target is found, GSAP animates the Rapier body translation to that target and calls onSnap.

The repair game uses this to place replacement parts and broken parts into case placeholders.

Hand-Controlled Grab

If handControlled is true, GrabbableObject also reads:

useHandTrackingSnapshot()

Hand grab flow:

1. Find a detected hand where hand.isFist is true.
2. Compute the visual center of the hand from landmark bounds.
3. Convert that screen-space point to a camera ray.
4. Raycast against the object.
5. Use a small set of offset rays around the center to make hit detection more forgiving.
6. If the object is in range and hit, enter handHolding.
7. Move the object toward a hold target in front of the camera while the fist remains closed.
8. When the fist opens or disappears, release and snap if possible.

This is an approximation, not a full 3D hand collider. It is a practical prototype compromise because MediaPipe gives normalized camera-space landmarks and relative depth, not stable world-space hand meshes.

Player Input Bridge

The player controller owns raw keyboard and mouse input:

src/world/player/PlayerController.tsx

It calls:

• interaction.pressInteract() when E is pressed and the focused handle is a trigger
• interaction.pressInteract() on mouse down when the focused handle is a grab
• interaction.releaseInteract() on mouse up when a grab is active

Input is ignored while:

• the settings menu is open
• a cinematic is playing

Movement lock is read separately from useRepairMovementLocked, which locks the player during focused repair steps.

UI Prompt

The prompt lives in:

src/components/ui/InteractPrompt.tsx

It appears only when:

• camera mode is player
• a focused interaction exists
• the player is not holding an object
• the focused interaction is a trigger

The prompt does not appear for grab objects, because grabs are mouse/hand actions rather than E trigger actions.

Debug Controls

Interaction debugging is split between:

• lil-gui Interaction folder for showing interaction spheres
• lil-gui GrabbableObject folder for grab tuning
• debug physics scene for live trigger/grab testing
• hand-tracking debug panel for hand grab state

Use:

http://localhost:5173/?debug

Then switch the scene mode to Physics from lil-gui.

Why This Architecture Works

The interaction layer separates concerns:

• R3F objects know their distance/raycast hit state.
• The player controller owns input events.
• UI only subscribes to a snapshot.
• Gameplay objects receive semantic callbacks like onTrigger, onSnap, or onPositionChange.

This keeps the repair game focused on gameplay rules instead of low-level input plumbing.

Known Limitations

• Only one focused handle is stored at a time.
• The focus rule is camera ray based, so side-facing interactions can feel strict without larger meshes or radii.
• Hand grab uses screen-space raycasts, not physical hand colliders.
• The manager is singleton-based, so tests must call destroy() or isolate state when needed.
• nearby is boolean, not a list exposed to UI, so the current UI cannot rank multiple nearby objects.