CLUSTER 3: VALIDATION

Introduction

The successful implementation of CCAM depends mainly on its acceptance and adoption in society. The decisive factor will be assuring the effective safety of CCAM. Therefore, the validation of the vehicles’ automated driving functions and of their operation in the intended ODD is forming an important cluster in the portfolio of activities of the CCAM Partnership.

Cluster 3 Objectives

• Validation methods applicable and recommendation of validation maturity necessary for FOTs and Living Labs, as described under “Large-scale demonstrations” above (to Cluster 1).
• Validation methodologies for application to vehicle technologies (to ensure the safe operation of CAVs and enable societal acceptance) (to Cluster 2).
• Enhanced methodologies and tools for integrating validated CCAM in the transport system (to Cluster 4).
• Requirements for validation methodology for cyber-security including protection against tampering of CCAM systems (to Cluster 5).

Higher levels of automated driving require scenario-based validation methodologies following hybrid approaches with physical and virtual testing. This is necessary to reduce the high number of test kilometres needed for safety validation specifically with more complex ODDs. The key R&I action in Cluster 3, “Future-proof methodologies and tools for validation”, addresses this need and also considers the possibility of updates of hardware and software during the whole vehicle lifecycle as well as self-learning capabilities of future automated driving systems. Validation in operation is an important aspect, also in view of the ageing and performance degradation of systems on the one hand and the perspective of lifetime compliance on the other hand. Next to the safety of the intended functionality, this R&I action addresses the functional safety of such systems and will incorporate the results from Cluster 5 with regard to the validation for cyber-security including protection against tampering. The R&I action “Future-proof methodologies and tools for validation” reflects the need to move more and more from a vehicle-centric perspective towards validation from the perspective of the transport system, including a multitude of other road users and the relevant infrastructure with its changes over time. While the focus is on validation for safety and security, this R&I action is open to address additional effects of CCAM, e.g., on traffic flow, which is facilitated by the adoption of this system perspective. It also includes the development of a commonly accepted and harmonised simulation environment for the virtual testing of CCAM functions and systems in their connectivity context.

A challenge for scenario-based validation will be identifying and considering all relevant critical scenarios with their criticality and probability of occurrence. This challenge is addressed by another R&I action in this cluster aiming at the establishment of an “EU wide database of relevant scenarios for validation”. The scope of this R&I action covers the derivation, definition and collection of safety critical scenarios for CCAM systems and their sharing as a basis for the verification, validation and assessment of CCAM systems in industrial development processes, in future type approval schemes and in consumer testing campaigns (like Euro NCAP). The action will build upon the “Future-proof methodologies and tools for validation” and will also make use of the “Test data exchange framework” from Cluster 7 as well as the “System architecture for data sharing” from Cluster 5. At the same time, the EU wide database of relevant scenarios for validation will be linked to the “EU wide knowledge base” to be developed in Cluster 7.

Validation will also need to take in into account human factors. This applies to human-technology interaction as well as to the understanding of human driving performance as a reference for the CCAM systems’ performance in reducing the number of fatalities and crashes. A specific R&I action developing a “Human reference for automated driving” validated by real-world tests will serve the latter purpose. Such a reference model of the performance spectrum of human drivers will allow a direct comparison to an automated driving system in the simulation of a specific situation and thus support the deployment of the results from the R&I action “Future-proof methodologies and tools for validation”. Together with results from Cluster 6 on societal and people needs, it will also form the basis of developing a common understanding of the required safety and reliability level of CCAM and support the definition of an acceptable behaviour of automated driving functions in mixed traffic.

The human reference model should be used in validation of CCAM by including driver behaviour as a parameter of scenarios in the EU wide database of relevant scenarios for validation. Taking into account human factors into validation methodologies would not only be beneficial for high levels of automation but would also strengthen the safety validation of lower levels such as ADAS.

Since safe human-vehicle interaction is an important element of the safety of highly automated driving systems, there is also a need for harmonised HMI designs as well as harmonised HMI testing and validation, which is addressed by a fourth R&I action in this cluster on “Specific HMI testing and validation procedures, methods and tools for higher levels of automation”. This applies both to driver vehicle interaction and to the interaction with other road users, for which new intuitive interaction principles will be needed, when the human driver is not involved in the control-loop of the vehicle. This R&I action will profit from close cooperation with the “HMI development for CCAM” in Cluster 2 (to ensure that development and validation methodologies are aligned) and complement the results from the R&I action on “Future-proof methodologies and tools for validation”.

While different assessment methods have been worked on for automated driving functions, and a mix of different homologation methods has been proposed, to date no common standard methodologies exist that meet all the requirements for testing, validation and certification of all levels and Use Cases of automated driving. The discussion on this is on-going regarding EU type vehicle approval rules as well as in the framework of the UNECE (WP.1 and WP.29). Regarding EU projects, the HEADstart and SUNRISE projects perform R&I activities related to Cluster 3.

Several national and European projects have already started to collect relevant scenarios for verification and validation and store them in databases. Database structures and related methodologies have been defined and implemented on the national level. A sharing framework for aggregated test data and a common data format for EU wide application have also been developed, amongst others by the L3Pilot project, the predecessor of Hi-Drive. There is, however, no EU wide database of relevant scenarios nor an agreed database structure.

Regarding the “Human reference for automated driving”, statistical data is available on the number of road crashes, injuries and road fatalities in many countries, which can be related to the number of personkilometres travelled to get an idea of overall human driving, vehicle and infrastructure performance in terms of safety. Statistical evidence, however, is missing on the performance spectrum of human drivers in the variety of specific situations which might be critical for automated driving systems. Several software modules to simulate human driver behaviour do exist today, but they only cover specific aspects of human driving performance and do not cover the full spectrum of drivers with statistical data on the probability of certain behavioural patterns. Towards this direction the i4Driving and the BERTHA project perform R&I activities related to Cluster 3.

The HMI design of highly automated vehicles is the subject of several R&I projects at the moment, addressing challenges such as those arising from mode transition situations. Although there is an agreement on the need for harmonised HMI designs for such vehicles and safe human-vehicle interaction is an important element of driving safety, there are no commonly accepted HMI testing and validation procedures, methods and tools available yet for higher levels of automation.

In addition to R&I, also regulatory activities on the validation of CCAM systems have been initiated, on the international level notably in the framework of the UNECE Working Party 29 on Automated/ Autonomous and Connected Vehicles (GRVA). Representatives of members in the CCAM Association are taking part in relevant UNECE meetings, thus making sure that R&I in the CCAM Partnership is aligned with the current state of discussions at UNECE. The EU is a pioneer in this regard with the European Commission implementing regulation (EU) 2022/1426 on highly automated, driverless vehicles. This is duly considered and reflected in this updated SRIA. The regulatory framework, for example, already foresees remote operation in the sense of an authorised person outside a driverless vehicle requesting its automated driving system to perform specific manoeuvres in particular situations. In contrast to that, remote driving would imply an operator taking full control and driving the vehicle directly through a dedicated interface, for instance to have the vehicle out of the way. The validation of such remote driving remains a research issue addressed in this SRIA update. UN Regulation R155 delivers uniform provisions concerning the approval of vehicles with regards to cyber security and cyber security management systems, but does not consider a verification testing methodology to confirm the effectiveness of the cyber security applied to the vehicles. The definition of standard test cases would further improve the cyber security of highly automated vehicles in particular.

Cluster 3 R&I Actions

The specific R&I actions, detailed also above, relating to Cluster 3 are the following:

• Future-proof methodologies and tools for validation. Common methodologies and tools (physical, virtual and hybrid) need to be developed to enable the verification and validation of CCAM systems in industrial development processes, in future type approval schemes and in consumer testing campaigns. This must consider maintenance, updates and upgrades of both software and hardware during the whole vehicle lifecycle and includes the development of a commonly accepted and harmonised simulation environment for the virtual testing of CCAM functions and systems. Advances in simulation tools and virtual models for realistic simulations are also needed, including digital twins. More and more, validation methodologies need to consider the context of the transport system, i.e. the interaction with other road users and the physical and digital infrastructure in the respective ODD. This will allow, amongst others, the validation of effects on traffic flow as well as the validation of coordinated CCAM and remote management concepts, including remote driving.
• EU wide database of relevant scenarios for validation. Future methodologies for the validation of CCAM systems will make use of scenario-based approaches. The scope of this R&I action covers the definition and collection of safety-critical scenarios for CCAM systems with their level of criticality and risk of occurrence, their sharing in an EU wide database and the derivation of test cases as a basis for the validation of CCAM systems in industrial development processes, in future type approval schemes and in consumer testing campaigns. The database shall be updated continuously and also reflect critical scenarios derived from in-service monitoring and reporting. Moreover, virtual simulations of road traffic need to be better understood as a source for the generation of relevant future scenarios for validation.
• Human Reference for Automated Driving. A fully validated and robust reference model of human behaviour depicting the performance spectrum of human drivers in critical situations will allow a direct comparison to an automated driving system in simulation and form the basis of developing a common understanding of the required safety and reliability level of CCAM including well-founded acceptance criteria. By not only representing an average driver behaviour, but covering the full performance spectrum, it will be possible to quantify the performance of an automated driving system in a specific situation in comparison to the human driver population. The reference model can then also be integrated in validation procedures and enable the inclusion of human behaviour as a parameter in validation scenarios.
• Specific HMI testing and validation procedures, methods and tools for higher levels of automation. Highly automated vehicles will need Human-Machine Interfaces (HMI) which allow intuitive and seamless transfer of control between the (on-board or remote) driver and the vehicle within short time as well as safe interaction with other road users. With the shared use of different vehicles becoming more common, and drivers/other road users having to adapt to the internal/external HMI of different vehicles, commonality of HMI principles and harmonisation of safety related HMI designs will be needed as well as their harmonised testing and validation.

Cluster 3 Expected Outcomes

• Common methodologies and tools defined, accepted and validated
  • by the CCAM value chain and its R&I partners for the efficient verification of CCAM systems in their R&I and product development processes,
  • by authorities and certification bodies for the validation of CCAM systems within type approval schemes and in future exemption procedures,
  • and by consumer testing campaigns for the safety rating of automated vehicles assisting users in identifying the safest choices for their needs.
• Verification, validation and rating procedures based on realistic and relevant test cases generated from an openly accessible and continuously updated EU wide database, providing the widest possible range of relevant scenarios which CCAM systems will potentially encounter on EU roads as a basis for robust system design.
• Extension of verification, validation and rating procedures to (self-)learning CCAM systems to maintenance, (over-the-air) updates and upgrades of hardware and software during the whole lifecycle of vehicles and infrastructure components as well as to coordinated CCAM and remote management including remote driving.
• Inclusion of the connectivity context in the verification, validation and rating procedures for CCAM systems which rely on connectivity for safety-critical functions, taking into account the aspect of cyber-security.
• A validated, robust and scalable reference model of human driving behaviour:
  • Replicating the full performance spectrum of human drivers, which allows comparing the performance of an automated driving system in a specific situation to the human driver population. This serves as a basis to define the required safety level of CCAM systems and to take decisions on validation requirements (acceptance criteria) in type approval schemes. The model will also help to define fair assessment criteria in consumer testing campaigns relative to human- driven vehicles and for the safety verification of CCAM systems in industrial development processes.
  • Serving as a reference for the automotive industry and its R&I partners to design human-like and therefore easily predictable and acceptable behaviour of automated driving functions in mixed traffic.
  • Applicable in virtual validation, verification and rating procedures and thereby helping the automotive industry, its R&I partners, certification bodies and consumer testing organisations to realistically represent the behaviour of other human-driven vehicles in the (virtual) simulation of mixed traffic. Virtual testing shortens development cycles and accelerates the implementation of CCAM technologies.
• Harmonised methods and tools for the testing, verification and validation of the internal and external HMI of highly automated vehicles defined and accepted by the CCAM value chain and its R&I partners as well as by authorities and certification bodies. These methods and tools will be based on common HMI principles and harmonised HMI designs.

Cluster 3 Timeline (as of Feb 2024)

The first three R&I actions of this cluster were planned to start early on. The actions in Cluster 3 will advance during the CCAM Partnership timeline towards implementation, particularly in Cluster 1 and in Cluster 2. While future-proof methodologies and tools for validation will already become available from a RIA project in the first phase of the CCAM Partnership, another RIA is foreseen to follow and to extent methodologies and tools for validation to self-learning CCAM systems as well as to over-the-air updates of CCAM functions during the whole vehicle lifecycle. Moreover, the connectivity context will need to be included for future CCAM systems relying on connectivity for safety-critical functions. The EU wide database of relevant scenarios will need to be updated and extended continuously according to extending ODDs and to the emergence of new traffic scenarios in the future road transport system. The R&I action on a human reference for automated driving will also deliver valuable results in the first phase of the CCAM Partnership by providing a human behavioural model validated for selected fields of application. Expanding these fields towards a fully validated human reference model replicating the full performance spectrum of human drivers in all relevant situations, however, will definitively require another RIA building on the results from the first phase. Research on specific HMI testing and validation procedures, methods and tools will start as an integral part of a RIA on future-proof methodologies and tools for validation. Based on common of HMI principles and the harmonisation of safety-related HMI designs from Cluster 2, another RIA on related testing and validation procedures, methods and tools will complete this research in the third phase of the CCAM Partnership.