Adaptive multimodal interaction in automated cars

Adaptive Multimodal interaction is key to enhance user experience, safety performance, and eventually build the trust in autonomous systems. Interfaces may vary depending on driving context (surrounding environment such as road traffic, weather, etc.), user’s cognitive and physical state, age and other situational factors.

This interaction is achieved by integrating various sensory modalities, such as visual displays, auditory feedback, and haptic sensations, dynamically adaptable to the above-mentioned factors. The combination of different sensors improves the driver situational awareness in case of takeover requests (TOR) and transitions between autonomous and manual driving modes. The multi-sensor HMI adaptation and its impact on user experience is under study within the AWARE2ALL EC project.

To this end, the project will use an immersive driving simulation platform providing a realistic 180° environment visualization in front of a real car cockpit. Inside the cockpit, different screens are synchronized with the simulation in real time (cluster, central display, HUD, and mirrors). It is also equipped with LEDs (side windows, windshield) and speakers (cockpit + headrest) to provide both visual and auditory feedback. New actuators and sensors, such as haptic feedback in the driver’s seat and steering wheel, will be installed in the project to enhance the demonstrator and user experience.

Metrics such as occupant’s health, real time workload estimations based on multi-sourced physiological data will be assessed, as well as indicators such as Driver Distraction, Hands-On-Steering wheel, and Situational Awareness level.

A large user testing campaign will be held to gather insights on adaptive multimodal interaction performance across various scenarios including different environmental conditions (weather, traffic, etc.) and user profiles to ensure inclusivity and effectiveness in real-world applications.

Source: The original article was published here