Situation-adaptive multimedia dialog system for automated vehicles (driving Assistance System) to convey the necessary situation awareness of the driver after a Take Over Request (TOR) and its legal safeguarding

Multimedia dialogue system (driving assistance system; 'Situation Awareness Manager') for use with the so-called Take Over Request (TOR) in vehicles. A recognition query is carried out and documented, and only if the driver answers correctly enough (situation awareness) is there a transfer of responsibility from the vehicle to the driver. Careless, routine responses are thus prevented.

• Multimedia, at least two types of communication are used
• Interactively guided system matching
• No careless, routine ticking off by the driver possible
• Handover procedure after TOR is documented in a legally binding manner
• Reduction of accident probability after TOR
• Increase in road safety, especially of automated vehicles

Automated vehicles at SAE level 3 (conditional driving automation) and level 4 (high driving automation).

From a technical point of view, automated driving is not a question of if, but only of when. In the next ten years, the share of automated vehicles will increase significantly. For example, the VW Group alone will invest around 30 billion euros in digitalisation and automation by the end of 2026. In addition to technical development, however, ergonomic requirements and legal issues must be answered, and insurers must also develop new concepts. The market launch for driving at SAE level 3 (conditional driving automation) is expected to take place before the end of 2022, for level 4 (high driving automation) probably between 2025 and 2027. The most dynamic development is assumed to be among Chinese manufacturers. The global market for automated vehicles was estimated at approximately USD 94 billion in 2021 and is expected to grow to over USD 1800 billion by 2030, with a compound annual growth rate (CAGR) of approximately 38% from 2021 to 2030.

When driving with vehicles at SAE level 3 (conditional driving automation) and level 4 (high driving automation), situations arise in which responsibility must or can be transferred from the vehicle back to the driver (Take Over Request, TOR). Here, for safety and legal reasons, it must be ensured that the driver has consciously and largely correctly grasped the current traffic situation/driving condition and is actually in a position to take over control of the vehicle (see, for example, "Legal issues in automated vehicles: critically considering the potential role of consent and ...",https://www.nature.com/articles/s41599-020-00644-2). If the driver is still distracted mentally or emotionally, or if he or she is negligent in the active detection of the external situation, the level of situational awareness achieved may be insufficient to ensure safe assumption of the driving task. The technical systems that are currently being considered or are available do not allow the driver to safely establish situational awareness because they do not provide optimal cognitive support and, above all, do not check whether all relevant information has been correctly recorded by the driver.

By means of the 'Situation Awareness Manager', a multimedia dialogue system, the transmission of information when a driver is requested to take over from an automated to a manual vehicle mode (Take Over Request, TOR) is cognitively more effective, because the information is transmitted in at least two different communication modes, e.g. visually and haptically. For example, a kind of checklist or small tasks concerning the situation for which the driver has to make a manual input can be displayed. If no input is made or the input is incorrect, the information transmission is repeated. The situation-adaptive multimodal dialogue system can, for example, also include gaze detection and release. The order and type of query by the system varies depending on the situation. In this way, the information can be weighted and a careless, routine ticking off of the checklist or solving of the tasks by the driver is prevented. The information perceived by the driver is then compared with the information transmitted. This results in an interactively guided system comparison of two cognitive states, namely of the driver and the vehicle. The perceptions observed by the driver with his sensory organs are matched with the environment, traffic and vehicle state detected by the sensor system. The system has a storage device in the sense of a black box that stores the sensor data, the transmitted information and the driver's perception. In summary, the system makes it possible to check the information recorded by the driver, i.e. a kind of a learning success check, in order to then document the handover procedure in a legally binding manner. In the event of a disputed accident situation after a driver takeover, it can then be proven whether and how, or to what extent, the driver has acquired the previously required situational awareness.

Further information on the technology:

https://www.iktd.uni-stuttgart.de/forschung/ide/sam