Adaptive Direct RGB-D Registration and Mapping for Large Motions

The mapping of environments is a prerequisite for many navigation and manipulation tasks. We propose a novel method for acquiring 3D maps of indoor scenes from a freely moving RGB-D camera. Our approach integrates color and depth cues seamlessly in a multi-resolution map representation. We consider measurement noise characteristics and exploit dense image neighborhood to rapidly extract maps from RGB-D images. An efficient ICP variant allows maps to be registered in real-time at VGA resolution on a CPU. For simultaneous localization and mapping, we extract key views and optimize the trajectory in a probabilistic framework. Finally, we propose an efficient randomized loop-closure technique that is designed for on-line operation. We benchmark our method on a publicly available RGB-D dataset and compare it with a state-of-the-art approach that uses sparse image features.

ISPRS Annals of Photogrammetry, Remote Sensing and Spatial Information Sciences, 2013

Registration of RGB-D data using visual features is often influenced by errors in the transformation of visual features to 3D space as well as the random error of individual 3D points. In a long sequence, these errors accumulate and lead to inaccurate and deformed point clouds, particularly in situations where loop closing is not feasible. We present an epipolar search method for accurate transformation of the keypoints from 2D to 3D space, and define weights for the 3D points based on the theoretical random error of depth measurements. Our results show that the epipolar search method results in more accurate 3D correspondences. We also demonstrate that weighting the 3D points improves the accuracy of sensor pose estimates along the trajectory.

Proceedings of the 9th International Conference on Computer Vision Theory and Applications, 2014

Visual mapping is a required capability for practical autonomous mobile robots where there exists a growing industry with applications ranging from the service to industrial sectors. Prior to map building, Visual Odometry(VO) is an essential step required in the process of pose graph construction. In this work, we first propose to tackle the pose estimation problem by using both photometric and geometric information in a direct RGBD image registration method. Secondly, the mapping problem is tackled with a pose graph representation, whereby, given a database of augmented visual spheres, a travelled trajectory with redundant information is pruned out to a skeletal pose graph. Both methods are evaluated with data acquired with a recently proposed omnidirectional RGBD sensor for indoor environments.

2018

Registration of multiple 3D data sets is a fundamental problem in many areas. Many researches and applications are using low-cost RGB-D sensors for 3D data acquisition. In general terms, the registration problem tries to find a transformation between two coordinate systems that better aligns the point sets. In order to review and describe the state-of-the-art of the rigid registration approaches, the authors decided to classify methods in coarse and fine. Due to the high variety of methods, they have made a study of the registration techniques, which could use RGB-D sensors in static scenarios. This chapter covers most of the expected aspects to consider when a registration technique has to be used with RGB-D sensors. Moreover, in order to establish a taxonomy of the different methods, the authors have classified those using different characteristics. As a result, they present a classification that aims to be a guide to help the researchers or practitioners to select a method based ...

Springer eBooks, 2021

Simultaneous Localization and Mapping (SLAM) is a critical system for Unmanned Aerial Vehicles (UAV), Autonomous Navigation (AN), Augmented Reality (AR) and Virtual Reality (VR). However, due to volume and computation limitations, the current SLAM systems on mobile devices are not robust enough for practical applications. In this paper, a new SLAM system is proposed, which can be applied to AR or VR. With the RGB-D input data, this SLAM system can produce 3D scene reconstruction and camera pose estimation in almost real time. The parameters of the system are optimized for use on mobile devices. In addition, an adaptive keyframe selection method is proposed to balance trade-off between accuracy and speed of the system. Experimental results demonstrated its superior performance in comparison with the state-ofthe-art methods in the same public benchmark dataset.

2012

This paper describes extensions to the Kintinuous algorithm for spatially extended KinectFusion, incorporating the following additions:(i) the integration of multiple 6DOF camera odometry estimation methods for robust tracking;(ii) a novel GPU-based implementation of an existing dense RGB-D visual odometry algorithm;(iii) advanced fused real-time surface coloring.

Pairwise alignment between point clouds is an important task in building 3D maps of indoor environments with partial information. The combination of 2D local features with depth information provided by RGB-D cameras are often used to improve such alignment. However, under varying lighting or low visual texture, indoor pairwise frame registration with sparse 2D local features is not a particularly robust method. In these conditions, features are hard to detect, thus leading to misalignment between consecutive pairs of frames. The use of 3D local features can be a solution as such features come from the 3D points themselves and are resistant to variations in visual texture and illumination. Because varying conditions in real indoor scenes are unavoidable, we propose a new framework to improve the pairwise frame alignment using an adaptive combination of sparse 2D and 3D features based on both the levels of geometric structure and visual texture contained in each scene. Experiments with datasets including unrestricted RGB-D camera motion and natural changes in illumination show that the proposed framework convincingly outperforms methods using 2D or 3D features separately, as reflected in better level of alignment accuracy.

2011 International Conference on Computer Vision, 2011

DTAM is a system for real-time camera tracking and reconstruction which relies not on feature extraction but dense, every pixel methods. As a single hand-held RGB camera flies over a static scene, we estimate detailed textured depth maps at selected keyframes to produce a surface patchwork with millions of vertices. We use the hundreds of images available in a video stream to improve the quality of a simple photometric data term, and minimise a global spatially regularised energy functional in a novel non-convex optimisation framework. Interleaved, we track the camera's 6DOF motion precisely by frame-rate whole image alignment against the entire dense model. Our algorithms are highly parallelisable throughout and DTAM achieves realtime performance using current commodity GPU hardware. We demonstrate that a dense model permits superior tracking performance under rapid motion compared to a state of the art method using features; and also show the additional usefulness of the dense model for real-time scene interaction in a physics-enhanced augmented reality application.

Sensors, 2014

The use of RGB-D sensors for mapping and recognition tasks in robotics or, in general, for virtual reconstruction has increased in recent years. The key aspect of these kinds of sensors is that they provide both depth and color information using the same device. In this paper, we present a comparative analysis of the most important methods used in the literature for the registration of subsequent RGB-D video frames in static scenarios. The analysis begins by explaining the characteristics of the registration problem, dividing it into two representative applications: scene modeling and object reconstruction. Then, a detailed experimentation is carried out to determine the behavior of the different methods depending on the application. For both applications, we used standard datasets and a new one built for object reconstruction.

arXiv (Cornell University), 2022

In this paper, we present a novel method for integrating 3D LiDAR depth measurements into the existing ORB-SLAM3 by building upon the RGB-D mode. We propose and compare two methods of depth map generation: conventional computer vision methods, namely an inverse dilation operation, and a supervised deep learning-based approach. We integrate the former directly into the ORB-SLAM3 framework by adding a so-called RGB-L (LiDAR) mode that directly reads LiDAR point clouds. The proposed methods are evaluated on the KITTI Odometry dataset and compared to each other and the standard ORB-SLAM3 stereo method. We demonstrate that, depending on the environment, advantages in trajectory accuracy and robustness can be achieved. Furthermore, we demonstrate that the runtime of the ORB-SLAM3 algorithm can be reduced by more than 40 % compared to the stereo mode. The related code for the ORB-SLAM3 RGB-L mode will be available as open-source software under https://github.com/ TUMFTM/ORB SLAM3 RGBL.