Enhanced Joint Trilateral Up-sampling for Super-Resolution

Computer Vision – ACCV 2016

Depth image super-resolution is an extremely challenging task due to the information loss in sub-sampling. Deep convolutional neural network have been widely applied to color image super-resolution. Quite surprisingly, this success has not been matched to depth super-resolution. This is mainly due to the inherent difference between color and depth images. In this paper, we bridge up the gap and extend the success of deep convolutional neural network to depth super-resolution. The proposed deep depth super-resolution method learns the mapping from a lowresolution depth image to a high resolution one in an end-toend style. Furthermore, to better regularize the learned depth map, we propose to exploit the depth field statistics and the local correlation between depth image and color image. These priors are integrated in an energy minimization formulation, where the deep neural network learns the unary term, the depth field statistics works as global model constraint and the colordepth correlation is utilized to enforce the local structure in depth images. Extensive experiments on various depth super-resolution benchmark datasets show that our method outperforms the stateof-the-art depth image super-resolution methods with a margin.

In this paper, we propose a convex optimization framework for simultaneous estimation of super-resolved depth map and images from a single moving camera. The pixel measurement error in 3D reconstruction is directly related to the resolution of the images at hand. In turn, even a small measurement error can cause significant errors in reconstructing 3D scene structure or camera pose. Therefore, enhancing image resolution can be an effective solution for securing the accuracy as well as the resolution of 3D reconstruction. In the proposed method, depth map estimation and image super-resolution are formulated in a single energy minimization framework with a convex function and solved efficiently by a first-order primal-dual algorithm. Explicit inter-frame pixel correspondences are not required for our super-resolution procedure, thus we can avoid a huge computation time and obtain improved depth map in the accuracy and resolution as well as highresolution images with reasonable time. The superiority of our algorithm is demonstrated by presenting the improved depth map accuracy, image super-resolution results, and camera pose estimation.

IEEE Transactions on Circuits and Systems for Video Technology

Deep convolutional neural network (DCNN) has been successfully applied to depth map super-resolution and outperforms existing methods by a wide margin. However, there still exist two major issues with these DCNN based depth map super-resolution methods that hinder the performance: i) The low-resolution depth maps either need to be up-sampled before feeding into the network or substantial deconvolution has to be used; and ii) The supervision (high-resolution depth maps) is only applied at the end of the network, thus it is difficult to handle large up-sampling factors, such as ×8, ×16. In this paper, we propose a new framework to tackle the above problems. First, we propose to represent the task of depth map super-resolution as a series of novel view synthesis sub-tasks. The novel view synthesis sub-task aims at generating (synthesizing) a depth map from different camera pose, which could be learned in parallel. Second, to handle large up-sampling factors, we present a deeply supervised network structure to enforce strong supervision in each stage of the network. Third, a multi-scale fusion strategy is proposed to effectively exploit the feature maps at different scales and handle the blocking effect. In this way, our proposed framework could deal with challenging depth map super-resolution efficiently under large up-sampling factors (e.g.×8, ×16). Our method only uses the low-resolution depth map as input, and the support of color image is not needed, which greatly reduces the restriction of our method. Extensive experiments on various benchmarking datasets demonstrate the superiority of our method over current state-of-the-art depth map super-resolution methods.

Pattern Recognition Letters, 2020

This paper proposes a new approach to single depth image super-resolution (SR), based upon a novel joint sparse coding model. A low-resolution color is used as a guide in the SR process.Firstly, we introduce synthetic characteristic image patch to learn a joint dictionary from the low-resolution depth map as well as its corresponding low-resolution intensity image. Then, we derive the joint nonlocal center sparse representation model based on sparse coding and theoretical analysis.In reconstruction process, we use Bayesian interpretation approach to estimation the sparse code coefficients for each unknown HR image patch. Meanwhile, we use an iterative algorithm to solve the JSC model. In addition, we exploit image patch redundancy within and across different scales, produce visually pleasing results without extensive training on external database. Experimental results demonstrate that the proposed method outperforms favorably many current state-of-the-art depth map super-resolution approaches on both visual effects and objective image quality and underpin the validity of our proposed model.

2012

ABSTRACT The recent development of low-cost and fast time-of-flight cameras enabled measuring depth information at video frame rates. Although these cameras provide invaluable information for many 3D applications, their imaging capabilities are very limited both in terms of resolution and noise level. In this paper, we present a novel method for obtaining a high resolution depth map from a pair of a low resolution depth map and a corresponding high resolution color image.

Symmetry

The super-resolution (SR) technique reconstructs a high-resolution image from single or multiple low-resolution images. SR has gained much attention over the past decade, as it has significant applications in our daily life. This paper provides a new technique of a single image super-resolution on true colored images. The key idea is to obtain the super-resolved image from observed low-resolution images. A proposed technique is based on both the wavelet and spatial domain-based algorithms by exploiting the advantages of both of the algorithms. A back projection with an iterative method is implemented to minimize the reconstruction error and for noise removal wavelet-based de-noising method is used. Previously, this technique has been followed for the grayscale images. In this proposed algorithm, the colored images are taken into account for super-resolution. The results of the proposed method have been examined both subjectively by observation of the results visually and objectively...

Applications of Digital Image Processing XXXII, 2009

In this paper we present a new method for superresolution of depth video sequences using high resolution color video. Here we assume that the depth sequence does not contain outlier points which can be present in the depth images. Our method is based on multiresolution decomposition, and uses multiple frames to search for a most similar depth segments to improve the resolution of the current frame. First step is the wavelet decomposition of both color and depth images. Scaling images of the depth wavelet decomposition, are superresolved using previous and future frames of the depth video sequence, due to their different nature. On the other side wavelet band are improved using both previous frames of the wavelet bands and wavelet bands of color images since similar edges might appear in both images. Our method shows significant improvements over some recent depth images interpolation methods.

proceedings of SPIE …, 2003

2005

A new approach for efficient and robust super-resolution is presented in this paper. The essence of the proposed method is in selectively applying a Bayesian MAP estimator to visually important image regions. Regions with high spatial activity are considered as visually important. The degree of image spatial activity is measured by calculating the gradient of the obtained estimate for the

IEEE Transactions on Image Processing, 2008

Super-resolution (SR) techniques produce a high-resolution image from a set of low-resolution undersampled images. In this paper, we propose a new method for super-resolution that uses sampling theory concepts to derive a noniterative SR algorithm. We first raise the issue of the validity of the data model usually assumed in SR, pointing out that it imposes a band-limited reconstructed image plus a certain type of noise. We propose a sampling theory framework with a prefiltering step that allows us to work with more general data models and also a specific new method for SR that uses Delaunay triangulation and B-splines to build the super-resolved image. The proposed method is noniterative and well posed. We prove its effectiveness against traditional iterative and noniterative SR methods on synthetic and real data. Additionally, we also prove that we can first solve the interpolation problem and then make the deblurring not only when the motion is translational but also when there are rotations and shifts and the imaging system Point Spread Function (PSF) is rotationally symmetric.