Definition: Gesture control is the operation of digital content with movements of the hands and fingers, without a controller, mouse or touchscreen. In XR headsets and at kiosk terminals, cameras capture the hands. Software recognizes defined gestures from this data, such as pointing, grabbing or pressing the thumb and index finger together. With these gestures, you can rotate, take apart and select 3D models.

English: Gesture Control, Hand Gesture Interaction
German: Gestensteuerung, Gestenerkennung

How It Works

The basis is hand tracking, a form of motion tracking: cameras in the headset or above the screen capture the hand, and a model calculates the position of the joints from it. The WebXR Hand Input Module (W3C) describes 25 joint points per hand for this. The software derives gestures from the pose:

  • Pinch: The thumb and index finger touch. This is the equivalent of a click. Apple Vision Pro combines it with gaze: the user looks at an element and taps the fingers together.
  • Grab: The full hand closes to grab an object and move it.
  • Poke: The index finger taps virtual buttons directly.
  • Point: A ray from the hand selects distant objects.

The open standard OpenXR supplies joint data (XR_EXT_hand_tracking) and standard gestures through extensions, so applications run on headsets from different manufacturers.

Importance in Mechanical Engineering

  • Trade show demos: Visitors put on a headset and grab objects immediately, without learning button assignments. There are no controllers that you must clean or charge between users. The headset itself still needs hygiene measures.
  • Touchless kiosk terminals: A camera on the screen recognizes hand movements in the air. Prospects rotate a machine model without touching a shared surface.
  • Service with an AR headset: Technicians scroll through instructions with gestures. With dirty hands or thick protective gloves, however, recognition becomes unreliable. Voice commands are then often the better addition.
  • Training and design review: Gestures are sufficient to view and explain. Where precision and haptic feedback are important, VR controllers keep the advantage.

Frequently Asked Questions (FAQ)

How reliable is gesture control?

Very reliable in good light and with a clear view of the hands. Hidden fingers, fast movements and gloves lower the recognition rate. Test with real users and real light conditions before you use it.

Do I need special hardware?

Current standalone headsets and mixed reality headsets recognize hands with their built-in cameras. For kiosk terminals, you can add camera and depth sensors.