Lensless Imaging: A computational renaissance

2019 IEEE/CVF International Conference on Computer Vision (ICCV), 2019

Recent advancements in fields like Internet of Things (IoT), augmented reality, etc. have led to an unprecedented demand for miniature cameras with low cost that can be integrated anywhere and can be used for distributed monitoring. Mask-based lensless imaging systems make such inexpensive and compact models realizable. However, reduction in the size and cost of these imagers comes at the expense of their image quality due to the high degree of multiplexing inherent in their design. In this paper, we present a method to obtain image reconstructions from mask-based lensless measurements that are more photorealistic than those currently available in the literature. We particularly focus on FlatCam [2], a lensless imager consisting of a coded mask placed over a bare CMOS sensor. Existing techniques for reconstructing FlatCam measurements suffer from several drawbacks including lower resolution and dynamic range than lens-based cameras. Our approach overcomes these drawbacks using a fully trainable non-iterative deep learning based model. Our approach is based on two stages: an inversion stage that maps the measurement into the space of intermediate reconstruction and a perceptual enhancement stage that improves this intermediate reconstruction based on perceptual and signal distortion metrics. Our proposed method is fast and produces photo-realistic reconstruction as demonstrated on many real and challenging scenes.

IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020

We demonstrate a versatile thin lensless camera with a designed phase-mask placed at sub-2 mm from an imaging CMOS sensor. Using wave optics and phase retrieval methods, we present a general-purpose framework to create phase-masks that achieve desired sharp point-spread-functions (PSFs) for desired camera thicknesses. From a single 2D encoded measurement, we show the reconstruction of high-resolution 2D images, computational refocusing, and 3D imaging. This ability is made possible by our proposed high-performance contour-based PSF. The heuristic contour-based PSF is designed using concepts in signal processing to achieve maximal information transfer to a bit-depth limited sensor. Due to the efficient coding, we can use fast linear methods for high-quality image reconstructions and switch to iterative nonlinear methods for higher fidelity reconstructions and 3D imaging.

Abstract We describe a theoretical framework for reversibly modulating 4D light fields with patterned masks in the optical path of a lens based camera. Based on this framework, we present a novel design to reconstruct the 4D light field from a 2D camera image without any additional lenses as required by previous light field cameras. The patterned mask attenuates light rays inside the camera instead of bending them, and the attenuation recoverably encodes the ray on the 2D sensor.

Journal of biomedical optics, 2017

A stable optical system is required to acquire a high-quality image. A motionless lensless setup is designated to obtain high-resolution and large field of view images. The sample is sequentially illuminated with multiple random phase patterns, and the recorded images are subtracted from the system calibration images correspondingly. The resultant images are propagated to the sample plane. The summation of all images yields a final image with resolution of ∼4 μm, field of view of ∼15 mm2, and better signal-to-noise ratio. This technique provides a compact, stable, and cost-effective optical system.

2018

The fields of digital image processing (DIP) and computational photography are ever growing with new focuses on coded aperture imaging and its real-world applications. Research has shown that coded apertures are far superior to traditional circular apertures for various tasks. A variety of coded aperture patterns have been proposed and developed over the years for use in various applications such as defocus deblurring, depth estimation and light field acquisition. Traditional coded aperture masks are constructed from static materials such as cardboard and cannot be altered once their shapes have been defined. These masks are then physically inserted into the aperture plane of a camera-lens system which makes swapping between different patterned masks difficult. This is undesirable as optimal aperture patterns differ depending on application, scene content or imaging conditions and thus would need to be changed quickly and frequently. This dissertation proposes the design and develop...

Computer Vision and Image Understanding, 2012

Many computational imaging applications involve manipulating the incoming light beam in the aperture and image planes. However, accessing the aperture, which conventionally stands inside the imaging lens, is still challenging. In this paper, we present an approach that allows access to the aperture plane and enables dynamic control of its transmissivity, position, and orientation. Specifically, we present two kinds of compound imaging systems (CIS), CIS1 and CIS2, to reposition the aperture in front of and behind the imaging lens respectively. CIS1 repositions the aperture plane in front of the imaging lens and enables the dynamic control of the light beam coming to the lens. This control is quite useful in panoramic imaging at the single viewpoint. CIS2 uses a rear-attached relay system (lens) to replace the aperture plane behind the imaging lens, and enables the dynamic control of the imaging light jointly formed by the imaging lens and the relay lens. In this way, the common imaging beam can be coded or split in the aperture plane to achieve many imaging functions, such as coded aperture imaging, high dynamic range (HDR) imaging and light field sampling. In addition, CIS2 repositions the aperture behind, instead of inside, the relay lens, which allows the employment of the optimized relay lens to preserve the high imaging quality. Finally, we present the physical implementations of CIS1 and CIS2, to demonstrate (1) their effectiveness in providing access to the aperture and (2) the advantages of aperture manipulation in computational imaging applications.

Optics Express, 2014

Objects imaged through thin scattering media can be reconstructed with the knowledge of the complex transmission function of the diffuser. We demonstrate image reconstruction of static and dynamic objects with numerical phase conjugation in a lensless setup. Data is acquired by single shot intensity capture of an object coherently illuminated and obscured by an inhomogeneous medium, i.e. light diffracted at a specimen is scattered by a polycarbonate diffuser and the resulting speckle field is recorded. As a preparational step, which has to be performed only one time before imaging, the complex speckle field diffracted by the diffuser to the camera chip is measured interferometrically, which allows to reconstruct the transmission function of the diffuser. After insertion of the specimen, the speckle field in the camera plane changes, and the complex field of the sample can be reconstructed from the new intensity distribution. After initial interferometric measurement of the diffuser field, the method is robust with respect to a subsequent misalignment of the diffuser. The method can be extended to image objects placed between a pair of thin scattering plates. Since the object information is contained in a single speckle intensity pattern, it is possible to image dynamic processes at video rate.

a href= http://arxiv. org/abs/physics/0306174, 2003

A solution to the inversion problem of scattering would offer aberration-free diffraction-limited 3D images without the resolution and depth-of-field limitations of lens-based tomographic systems. Powerful algorithms are increasingly being used to act as lenses to form such images. Current image reconstruction methods, however, require the knowledge of the shape of the object and the low spatial frequencies unavoidably lost in experiments. Diffractive imaging has thus previously been used to increase the resolution ...

Compressive Sensing III, 2014

In certain imaging applications, conventional lens technology is constrained by the lack of materials which can effectively focus the radiation within reasonable weight and volume. One solution is to use coded apertures-opaque plates perforated with multiple pinhole-like openings. If the openings are arranged in an appropriate pattern, the images can be decoded, and a clear image computed. Recently, computational imaging and the search for means of producing programmable software-defined optics have revived interest in coded apertures. The former state-of-the-art masks, MURAs (Modified Uniformly Redundant Arrays) are effective for compact objects against uniform backgrounds, but have substantial drawbacks for extended scenes: 1) MURAs present an inherently ill-posed inversion problem that is unmanageable for large images, and 2) they are susceptible to diffraction: a diffracted MURA is no longer a MURA. This paper presents a new class of coded apertures, Separable Doubly-Toeplitz masks, which are efficiently decodable, even for very large images-orders of magnitude faster than MURAs, and which remain decodable when diffracted. We implemented the masks using programmable spatial-lightmodulators. Imaging experiments confirmed the effectiveness of Separable Doubly-Toeplitz masksimages collected in natural light of extended outdoor scenes are rendered clearly.

2014

The open-source Robot Operating System (ROS) is a mixed and scalable P2P network-based robotics framework. We examine lensless compressive imaging using a hardware apparatus assembly having an imaging sensor, but no lens. Cameras with lenses have been the standard, but several factors constrain their application. Lensless cameras may reduce the cost, size, and weight of image processing as we move away from use of expensive lenses in robot designs. Lensless cameras can be used also in applications such medicine, where apparatus size is very important. To support our objective we show how ROS applications are developed and most importantly how one can build applications that allows users to complete useful tasks in a timely manner with high performance. iii ACKNOWLEDGMENTS I would like to thank my commitee members Professors Jayantha Herath, Yi Zhang and Andrew Anda for their continuous support. I have to thank my project sponsor Stone3000 Inc. Stone3000 inc. financed this project, provided material; and provided the lab environment needed for my experiments. Special thanks to St. Cloud State University, Computer Science Department and faculty, Electrical Engineering Department and faculty, Ms. Mary Thrall, fellow classmates and friends for providing a great learning atmosphere and for providing skills that made a big contribution during this project. Thanks to my family Andjela, Stefan and Annamaria, for being there when I needed them. iv As per NDA agreements with Stone3000 and optical sensor manufacturer Aptina Inc. the codes and data sheets used in these experiments are considered proprietary and are not discussed in this paper. v