Evaluating Dynamic Environment Difficulty for Collision Avoidance Benchmarking

Delft University of Technology

A robot navigating through the environment with low difficulty level.

A robot navigating through the environment with high difficulty level.

Abstract

Dynamic obstacle avoidance is very important for autonomous systems like Micro Aerial Vehicles (MAVs) and service robots. With numerous methods proposed to solve this challenge, it becomes imperative to benchmark these methods by testing them within different maps. During the benchmarking process, a critical aspect often overlooked is the difficulty of the maps used for experiments. Mentioning the difficulty is important not only for a fair comparison between different methods but also for a comprehensive analysis of the performance of one method. Currently, there is no quantitative metric to quantify the difficulty of a dynamic environment. In this paper, we propose several metrics to measure the difficulty of dynamic environments and validate them through over 1.5 million experiment trials. Finally, the survivability metric is chosen. Its reliability is underscored by the correlation with the success rates of typical collision avoidance methods with a Spearman's Rank Correlation Coefficient (SRCC) of over 0.9. This metric not only facilitates fair benchmarking but also provides insights for refining collision avoidance methods, furthering the evolution of autonomous systems.

Obstacle Environment Generator





Video Presentation