The Stixel World: A medium-level representation of traffic scenes

2016 IEEE Intelligent Vehicles Symposium (IV), 2016

In this paper we present Semantic Stixels, a novel vision-based scene model geared towards automated driving. Our model jointly infers the geometric and semantic layout of a scene and provides a compact yet rich abstraction of both cues using Stixels as primitive elements. Geometric information is incorporated into our model in terms of pixel-level disparity maps derived from stereo vision. For semantics, we leverage a modern deep learning-based scene labeling approach that provides an object class label for each pixel. Our experiments involve an in-depth analysis and a comprehensive assessment of the constituent parts of our approach using three public benchmark datasets. We evaluate the geometric and semantic accuracy of our model and analyze the underlying run-times and the complexity of the obtained representation. Our results indicate that the joint treatment of both cues on the Semantic Stixel level yields a highly compact environment representation while maintaining an accuracy comparable to the two individual pixel-level input data sources. Moreover, our framework compares favorably to related approaches in terms of computational costs and operates in real-time.

IEEE Access

Autonomous mobile robots use computational techniques of great complexity so that to allow navigation in various types of dynamic environments, avoiding collisions with obstacles and always seeking to optimize the best route, ultimately enabling them to operate in a safe and precise manner. In order for navigation at this level to be possible, a variety of computer vision and intelligent sensing techniques are used. The potential of an intelligent computer vision system to detect and predict the actions of dynamic agents on the streets is applied to increase traffic safety with intelligent robotic vehicles. In this paper we present a systematic review of computer vision models for the detection and tracking of obstacles in traffic environments. Specifically, we cover works involving 2D and 3D (stereo vision) data fusion for both internal and external perception, as well as current trends regarding efficient model design and temporallyaware architectures. We provide a thorough discussion on the main positive and negative points of the state-of-the-art in Visual Robotic Attention, as well as share our experience and contributions in applying visual perception for external obstacle detection and tracking, and internal (driver) monitoring. The results presented should serve as a compilation of the history of visual perception for autonomous mobile robots (specifically, Advanced Driver Assistance Systems (ADAS) and Autonomous Vehicles), thus providing the reader with a comprehensive basis on both the main contributions and the state-of-the-art in the field. INDEX TERMS Autonomous vehicles, computer vision, deep learning, obstacle detection and classification.

IEEE Transactions on Intelligent Transportation Systems, 2022

The European UP-Drive project addresses transportation-related challenges by providing key contributions that enable fully automated vehicle navigation and parking in complex urban areas, which results in a safer, inclusive, affordable and environmentally friendly transportation system. For this purpose, the project consortium developed a prototype electrical vehicle equipped with cameras and LiDARs sensors that is capable to autonomously drive around the city and find available parking spots. In UP-Drive, we created an accurate, robust and redundant multi-modal environment perception system that provides 360 • coverage around the vehicle. This paper summarizes the work of the project related to the surround view semantic perception using fisheye and narrow field-of-view semantic virtual cameras. Deep learning-based semantic, instance and panoptic segmentation networks, which satisfy requirements in accuracy and efficiency have been developed and integrated into the final prototype. The UP-Drive automated vehicle has been successfully demonstrated in urban areas after extensive experiments and numerous field tests.

Proceedings of the 6th Workshop on Parallel Programming and Run-Time Management Techniques for Many-core Architectures - PARMA-DITAM '15, 2015

Intelligent vehicles (IVs) need a perception system to model the surrounding environment. The Hybrid Sampling Bayesian Occupancy Filter (HSBOF) is a perception algorithm monitoring a grid-based model of the environment called "occupancy grid". It is a highly data-parallel algorithm and requires a high computational performance to be executed in reasonable time. It is currently implemented in CUDA on a NVIDIA GPU. However, the GPU is power consuming and its purchase cost is too high for the embedded market. In this paper, we prove that, the couple embedded many-core/OpenCL is a feasible hardware/software architecture for replacing the GPU/CUDA. Our OpenCL implementation and experimental results on a testing hardware showed that a many-core can produce an occupancy grid every 168ms while consuming 40 times less power than the GPU. The results are promising for a future integration into IVs.

Handbook of Intelligent Vehicles, 2012

This chapter describes detection and tracking of moving objects (DATMO) for purposes of autonomous driving. DATMO provides awareness of road scene participants, which is important in order to make safe driving decisions and abide by the rules of the road. Three main classes of DATMO approaches are identified and discussed. First is the traditional approach, which includes data segmentation, data association, and filtering using primarily Kalman filters. Recent work within this class of approaches has focused on pattern recognition techniques. The second class is the model-based approach, which performs inference directly on the sensor data without segmentation and association steps. This approach utilizes geometric object models and relies on non-parametric filters for inference. Finally, the third class is the grid-based approach, which starts by constructing a low level grid representation of the dynamic environment. The resulting representation is immediately useful for determining free navigable space within the dynamic environment. Grid construction can be followed by segmentation, association, and filtering steps to provide object level representation of the scene. The chapter introduces main concepts, reviews relevant sensor technologies, and provides extensive references to recent work in the field. The chapter also provides a taxonomy of DATMO applications based on road scene environment and outlines requirements for each application.

2021 IEEE Intelligent Vehicles Symposium (IV), 2021

International Journal of Computer Vision

This work presents and evaluates a novel compact scene representation based on Stixels that infers geometric and semantic information. Our approach overcomes the previous rather restrictive geometric assumptions for Stixels by introducing a novel depth model to account for non-flat roads and slanted objects. Both semantic and depth cues are used jointly to infer the scene representation in a sound global energy minimization formulation. Furthermore, a novel approximation scheme is introduced in order to significantly reduce the computational complexity of the Stixel algorithm, and then achieve real-time computation capabilities. The idea is to first perform an over-segmentation of the image, discarding the unlikely Stixel cuts, and apply the algorithm only on the remaining Stixel cuts. This work presents a novel over-segmentation strategy based on a fully convolutional network, which outperforms an approach based on using local extrema of the disparity map. We evaluate the proposed ...

SSRN Electronic Journal

Overview of environment perception for intelligent vehicles supposes to the state-of-the-art algorithms and modeling methods are given, with a summary of their pros and cons. A special attention is paid to methods for lane and road detection, traffic sign recognition, vehicle tracking, behavior analysis, and scene understanding. Integrated lane and vehicle tracking for driver assistance system that improves on the performance of both lane tracking and vehicle tracking modules. Without specific hardware and software optimizations, the fully implemented system runs at near-real-time speeds of 11 frames per second. On-road vision-based vehicle detection, tracking, and behavior understanding. Vision based vehicle detection in the context of sensor-based on-road surround analysis. We detail advances in vehicle detection, discussing monocular, stereo vision, and active sensor-vision fusion for on-road vehicle detection. The traffic sign detection detailing detection systems for traffic sign recognition (TSR) for driver assistance. Inherently in traffic sign detection to the various stages: segmentation, feature extraction, and final sign detection.

Mechatronics, 2003

The Navlab group at Carnegie Mellon University has a long history of development of automated vehicles and intelligent systems for driver assistance. The earlier work of the group concentrated on road following, cross-country driving, and obstacle detection. The new focus is on short-range sensing, to look all around the vehicle for safe driving. The current system uses video sensing, laser rangefinders, a novel light-stripe rangefinder, software to process each sensor individually, a map-based fusion system, and a probability based predictive model. The complete system has been demonstrated on the Navlab 11 vehicle for monitoring the environment of a vehicle driving through a cluttered urban environment, detecting and tracking fixed objects, moving objects, pedestrians, curbs, and roads.

2008

This paper describes the intelligence components associated with the system design developed for Team Gator Nation's submission to the 2007 DARPA Urban Challenge. In this event, vehicles had to navigate on city streets while obeying basic traffic laws. One of the major challenges was interacting with other vehicles such as at intersections. To address these challenges, a hybrid Toyota Highlander was automated and instrumented with pose estimation (GPS and inertial) and object detection (vision and ladar) sensors. A control architecture was developed which integrates planning, perception, decision making, and control elements. The intelligence element implements the Adaptive Planning Framework which was developed by researchers at the University of Florida. This framework provides a means for situation assessment, behavior mode evaluation, and behavior selection and execution. The paper describes this architecture and concludes with lessons learned from participation in the Urba...