Technology Offers

Situation-adaptive multimedia dialog system for automated vehicles (driving Assistance System) to convey the necessary situation awareness of the driver after a Take Over Request (TOR) and its legal safeguarding

Abstract

Multimedia dialogue system (driving assistance system; 'Situation Awareness Manager') for use with the so-called Take Over Request (TOR) in vehicles that enable improved driver take-over after automated driving at SAE level 3 (conditional driving automation) and level 4 (high driving automation). Through the dialogue system, the perceptions made by the driver are brought into line with the traffic state around the vehicle detected by the sensor system (comparison of the detection results of camera, radar, lidar, ultrasonic sensors - so-called sensor fusion) in order to achieve a level of information necessary for manual driving. A recognition query is carried out and documented, and only if the driver answers correctly enough (situation awareness) is there a transfer of responsibility from the vehicle to the driver. Careless, routine responses are thus prevented.

Background

From a technical point of view, automated driving is not a question of if, but only of when. In the next ten years, the share of automated vehicles will increase significantly. For example, the VW Group alone will invest around 30 billion euros in digitalisation and automation by the end of 2026. In addition to technical development, however, ergonomic requirements and legal issues must be answered, and insurers must also develop new concepts. The market launch for driving at SAE level 3 (conditional driving automation) is expected to take place before the end of 2022, for level 4 (high driving automation) probably between 2025 and 2027. The most dynamic development is assumed to be among Chinese manufacturers. The global market for automated vehicles was estimated at approximately USD 94 billion in 2021 and is expected to grow to over USD 1800 billion by 2030, with a compound annual growth rate (CAGR) of approximately 38% from 2021 to 2030.

Problem

When driving with vehicles at SAE level 3 (conditional driving automation) and level 4 (high driving automation), situations arise in which responsibility must or can be transferred from the vehicle back to the driver (Take Over Request, TOR). Here, for safety and legal reasons, it must be ensured that the driver has consciously and largely correctly grasped the current traffic situation/driving condition and is actually in a position to take over control of the vehicle (see, for example, "Legal issues in automated vehicles: critically considering the potential role of consent and ...",https://www.nature.com/articles/s41599-020-00644-2). If the driver is still distracted mentally or emotionally, or if he or she is negligent in the active detection of the external situation, the level of situational awareness achieved may be insufficient to ensure safe assumption of the driving task. The technical systems that are currently being considered or are available do not allow the driver to safely establish situational awareness because they do not provide optimal cognitive support and, above all, do not check whether all relevant information has been correctly recorded by the driver.

Solution

By means of the 'Situation Awareness Manager', a multimedia dialogue system, the transmission of information when a driver is requested to take over from an automated to a manual vehicle mode (Take Over Request, TOR) is cognitively more effective, because the information is transmitted in at least two different communication modes, e.g. visually and haptically. For example, a kind of checklist or small tasks concerning the situation for which the driver has to make a manual input can be displayed. If no input is made or the input is incorrect, the information transmission is repeated. The situation-adaptive multimodal dialogue system can, for example, also include gaze detection and release. The order and type of query by the system varies depending on the situation. In this way, the information can be weighted and a careless, routine ticking off of the checklist or solving of the tasks by the driver is prevented. The information perceived by the driver is then compared with the information transmitted. This results in an interactively guided system comparison of two cognitive states, namely of the driver and the vehicle. The perceptions observed by the driver with his sensory organs are matched with the environment, traffic and vehicle state detected by the sensor system. The system has a storage device in the sense of a black box that stores the sensor data, the transmitted information and the driver's perception. In summary, the system makes it possible to check the information recorded by the driver, i.e. a kind of a learning success check, in order to then document the handover procedure in a legally binding manner. In the event of a disputed accident situation after a driver takeover, it can then be proven whether and how, or to what extent, the driver has acquired the previously required situational awareness.

Advantages

  • Situationally adaptive guidance
  • Multimedia, at least two types of communication are used
  • Sequence / type of query by the system varies depending on the situation
  • Weighting of traffic conditions and information
  • Interactively guided system matching
  • No careless, routine ticking off by the driver possible
  • Scope of the driver's situational awareness is recorded
  • Integrated storage device / black box
  • Handover procedure after TOR is documented in a legally binding manner
  • Reduction of accident probability after TOR
  • Increase in road safety, especially of automated vehicles

Fields of Application

Automated vehicles at SAE level 3 (conditional driving automation) and level 4 (high driving automation)

Exposé
Contact
Dr. Frank Schlotter
TLB GmbH
Ettlinger Straße 25
76137 Karlsruhe | Germany
Phone +49 721-79004-0
schlotter(at)tlb.de | www.tlb.de
Development Status
TRL2
Patent Situation
DE 102021209251 pending
Reference ID
21/018TLB
Service
Technologie-Lizenz-Büro GmbH is in charge of the exploitation of the technology and offers companies the possibility of cooperation and licensing.