PREPER – A Dataset for Safety Evaluation of AI Perception Systems

Deep learning algorithms are becoming the state-of-the-art technology used in the perception functions of AD and ADAS. With the safety assurance of level 5 AD systems the perception of the environment and especially dynamic objects becomes a central component. Data-driven AI algorithms are to a large extent determined by the training dataset. Interpretability and reliability of these algorithms are known to be challenging due to their data-driven nature.

AI based perception systems need to be tested and validated. Edge cases for testing can be derived from real world scenarios that lead to accidents caused by human drivers. Available public data is limited to normal driving situations, like KITTI http://www.cvlibs.net/datasets/kitti/ or the Waymo Open Dataset https://waymo.com/open/. Data that covers edge cases like safety critical scenarios to test these systems are still lacking.

PREPER (“PRE-crash PERception”) aims at providing a dataset for evaluation and benchmarking of the safety performance of object detection in AI driven visual perception systems for road vehicles. It is comprised of tests conducted at the AstaZero test track located in Västra Götaland, Sweden during December 2020 and January 2021. The test track and driving days were funded by the OpenResearch programme by Chalmers, RISE, and SAFER and the research and data preparation work was part of the FFI project “Safety-driven data labelling platform to enable safe and responsible AI”. The main data collection car (ego vehicle) was provided by the laboratory of Resource for Vehicle Research at Chalmers (REVERE).

All sequences in PREPER mimic pre-crash phases of two passenger cars, the ego car (REVERE) and the target car (AstaZero). To cover a variety of pre-crash scenarios the tests were conducted at the rural and city areas of the AstaZero track. In total there were eight different scenarios each with several variations of sequences that replicate the pre-crash phase of an accident situation. The selected scenarios are based on accident data analysis, EuroNCAP and U.S. NCAP test configurations as well as real world accident pre-crash data from the IGLAD pre-crash matrix http://www.iglad.net ensuring relevance for real life safety. Variations of the different sequences within a scenario include:

• With and without “collision” as a reference
• Fake/critical only maneuvers
• Occlusion by another car or a building
• Different degrees of “collision” overlap
• Different speeds of the target car

Additionally, there was a leading car in some sequences for occlusion purposes of the target car. In order to safely conduct the tests the speed of the vehicles was lower when driving the tests than it is in the replicated pre-crash scenarios in the PREPER data set. All sensor parameters like timestamps, camera frame rates, lidar rotation, and speeds are transformed to a unified and warped speed afterwards which reflects the real world speed of the pre-crash scenarios. The tests were driven by professional drivers from AstaZero to ensure a safe test performance. The warped speeds of the ego vehicle in rural scenarios is 80 kph and in the city 50 kph. The warped speed of the target car varies between the sequences. Ground truth for the leading car is not provided as it is not part of the benchmark.

This is an overview of the scenarios and the number of sequences in each scenario:

Test Area	Scenario	Sequences
rural	straight	29
rural	curve_left	31
rural	curve_right	30
city	cross1	5
city	cross2	6
city	cross3	10
city	cross4	8
city	turn	41
total		160

The PREPER dataset can be downloaded from https://github.com/AsymptoticAI/PREPER. The main dataset is made available as one archive containing the camera data and target poses along with a python script to calculate the KPIs. The optional lidar data is stored in a separate file.

For feedback, questions or if you like to provide benchmark results please contact jorg.bakker@asymptotic.ai