Multiverse Computing partnered with a major European aerospace OEM to develop a robust gesture recognition system for pilot hand gesture control inside aircraft cockpits. The project delivered a real-time computer vision pipeline that performs reliably with gloved and barehanded operation under varying cockpit lighting conditions, opening a new path for natural, hands-on-instrument interaction between pilot and aircraft.
The Challenge
The client required a gesture recognition system for pilot hand gesture control in aircraft cockpits, capable of working reliably with gloved hands, barehanded operation and varying lighting conditions. Standard hand-landmark approaches break down with gloved hands, which is the default operational state for the target use case. The mission demanded a custom computer vision approach that could recognize 25 static and dynamic gestures in real time and run inside cockpit hardware constraints.
Our Solution
Multiverse Computing built a custom computer vision pipeline based on a state-of-the-art object detection model (YOLOv11s), trained on a purpose-built dataset of more than 11,000 manually labelled images covering both gloved and barehanded gestures. Our optimization technology shaped the resulting Multiverse Computing model for in-cockpit deployment, and the pipeline was packaged with a desktop application providing real-time inference, a configurable detection panel and a camera feed for live operation.
- Custom YOLOv11s object detection pipeline tailored to cockpit gesture recognition.
- Purpose-built dataset of more than 11,000 manually labelled images covering gloved and barehanded operation.
- 25 static and dynamic gestures defined for full cockpit coverage.
- Desktop inference application with real-time camera feed and configurable hyperparameters.
- Model optimization for in-cockpit deployment, ready to integrate into cockpit hardware.
Results
Robust pilot gesture recognition under gloved and barehanded operation
90% accuracy on the initial gesture set
25 static and dynamic gestures defined for cockpit coverage
Purpose-built dataset of more than 11,000 manually labelled images
Real-time inference ready for integration into cockpit hardware
Reusable gesture recognition framework for other gloved-hand interaction scenarios
Strategic outcome: a robust, real-time cockpit gesture recognition capability that handles gloved and barehanded operation, scales to a full 25-gesture set, and gives the client a reusable computer vision framework for natural human-machine interaction in aviation.
