Lensless photography with only an image sensor

2014

We describe the optical, mathematical and computational foundations for a new class of lensless, ultra-miniature computational imagers and image sensors. Such sensors employ phase gratings that have provably optimal optical properties and are integrated with CMOS photodetector matrices. These imagers have no lens and can thus be made extremely small (∼100 µm) and very inexpensive (a few Euro cents). Because the apertures are small, they have an effective depth of field ranging from roughly 1 mm to infinity. The grating acts as a two-dimensional visual "chirp" and preserves image power throughout the Fourier plane; thus the captured signals preserve image information. The final digital image is not captured as in a traditional camera but is instead computed from raw photodetector signals. The novel representation at the photodetectors demands powerful algorithms such as deconvolution, Bayesian estimation, or matrix inversion with Tikhonov regularization be used to compute the image, each having different bandwidth, space and computational complexities for a given image fidelity. Such imaging architectures can also be tailored to extract application-specific information or compute decisions (rather than compute an image) based on the optical signal. In most cases, both the phase grating and the signal processing can incorporate prior information about the visual field and the imaging or estimation task at hand. Our sensor design methodology relies on modular parallel and computationally efficient software tools for simulating optical diffraction, for CAD design and layout of gratings themselves, and for sensor signal processing. These sensors are so small they should find use in endoscopy, medical sensing, machine inspection, surveillance and the Internet of Things, and are so inexpensive that they should find use in distributed network applications and in a number of single-use or disposable applications, for instance in military, hazardous natural and industrial conditions.

Abstract We describe a theoretical framework for reversibly modulating 4D light fields with patterned masks in the optical path of a lens based camera. Based on this framework, we present a novel design to reconstruct the 4D light field from a 2D camera image without any additional lenses as required by previous light field cameras. The patterned mask attenuates light rays inside the camera instead of bending them, and the attenuation recoverably encodes the ray on the 2D sensor.

Compressive Sensing III, 2014

In certain imaging applications, conventional lens technology is constrained by the lack of materials which can effectively focus the radiation within reasonable weight and volume. One solution is to use coded apertures-opaque plates perforated with multiple pinhole-like openings. If the openings are arranged in an appropriate pattern, the images can be decoded, and a clear image computed. Recently, computational imaging and the search for means of producing programmable software-defined optics have revived interest in coded apertures. The former state-of-the-art masks, MURAs (Modified Uniformly Redundant Arrays) are effective for compact objects against uniform backgrounds, but have substantial drawbacks for extended scenes: 1) MURAs present an inherently ill-posed inversion problem that is unmanageable for large images, and 2) they are susceptible to diffraction: a diffracted MURA is no longer a MURA. This paper presents a new class of coded apertures, Separable Doubly-Toeplitz masks, which are efficiently decodable, even for very large images-orders of magnitude faster than MURAs, and which remain decodable when diffracted. We implemented the masks using programmable spatial-lightmodulators. Imaging experiments confirmed the effectiveness of Separable Doubly-Toeplitz masksimages collected in natural light of extended outdoor scenes are rendered clearly.

2021

Lensless cameras provide a framework to build thin imaging systems by replacing the lens in a conventional camera with an amplitude or phase mask near the sensor. Existing methods for lensless imaging can recover the depth and intensity of the scene, but they require solving computationally-expensive inverse problems. Furthermore, existing methods struggle to recover dense scenes with large depth variations. In this paper, we propose a lensless imaging system that captures a small number of measurements using different patterns on a programmable mask. In this context, we make three contributions. First, we present a fast recovery algorithm to recover textures on a fixed number of depth planes in the scene. Second, we consider the mask design problem, for programmable lensless cameras, and provide a design template for optimizing the mask patterns with the goal of improving depth estimation. Third, we use a refinement network as a post-processing step to identify and remove artifacts...

2021

Imaging systems have long been designed in separated steps: the experience-driven optical design followed by sophisticated image processing. Such a general-propose approach achieves success in the past but left the question open for specific tasks and the best compromise between optics and post-processing, as well as minimizing costs. Driven from this, a series of works are proposed to bring the imaging system design into end-to-end fashion step by step, from joint optics design, point spread function (PSF) optimization, phase map optimization to a general end-to-end complex lens camera. To demonstrate the joint optics application with image recovery, we applied it to flat lens imaging with a large field of view (LFOV). In applying a super-resolution single-photon avalanche diode (SPAD) camera, the PSF encoded by diffractive op tical element (DOE) is optimized together with the post-processing, which brings the optics design into the end-to-end stage. Expanding to color imaging, opt...

Journal of biomedical optics, 2017

A stable optical system is required to acquire a high-quality image. A motionless lensless setup is designated to obtain high-resolution and large field of view images. The sample is sequentially illuminated with multiple random phase patterns, and the recorded images are subtracted from the system calibration images correspondingly. The resultant images are propagated to the sample plane. The summation of all images yields a final image with resolution of ∼4 μm, field of view of ∼15 mm2, and better signal-to-noise ratio. This technique provides a compact, stable, and cost-effective optical system.

2007

Abstract We describe a theoretical framework for reversibly modulating 4D light fields using an attenuating mask in the optical path of a lens based camera. Based on this framework, we present a novel design to reconstruct the 4D light field from a 2D camera image without any additional refractive elements as required by previous light field cameras. The patterned mask attenuates light rays inside the camera instead of bending them, and the attenuation recoverably encodes the rays on the 2D sensor.

Computer Vision and Image Understanding, 2012

Many computational imaging applications involve manipulating the incoming light beam in the aperture and image planes. However, accessing the aperture, which conventionally stands inside the imaging lens, is still challenging. In this paper, we present an approach that allows access to the aperture plane and enables dynamic control of its transmissivity, position, and orientation. Specifically, we present two kinds of compound imaging systems (CIS), CIS1 and CIS2, to reposition the aperture in front of and behind the imaging lens respectively. CIS1 repositions the aperture plane in front of the imaging lens and enables the dynamic control of the light beam coming to the lens. This control is quite useful in panoramic imaging at the single viewpoint. CIS2 uses a rear-attached relay system (lens) to replace the aperture plane behind the imaging lens, and enables the dynamic control of the imaging light jointly formed by the imaging lens and the relay lens. In this way, the common imaging beam can be coded or split in the aperture plane to achieve many imaging functions, such as coded aperture imaging, high dynamic range (HDR) imaging and light field sampling. In addition, CIS2 repositions the aperture behind, instead of inside, the relay lens, which allows the employment of the optimized relay lens to preserve the high imaging quality. Finally, we present the physical implementations of CIS1 and CIS2, to demonstrate (1) their effectiveness in providing access to the aperture and (2) the advantages of aperture manipulation in computational imaging applications.

Optics Express, 2015

Diffractive optical elements (DOE) show great promise for imaging optics that are thinner and more lightweight than conventional refractive lenses while preserving their light efficiency. Unfortunately, severe spectral dispersion currently limits the use of DOEs in consumer-level lens design. In this article, we jointly design lightweight diffractive-refractive optics and post-processing algorithms to enable imaging under white light illumination. Using the Fresnel lens as a general platform, we show three phase-plate designs, including a super-thin stacked plate design, a diffractive-refractive-hybrid lens, and a phase coded-aperture lens. Combined with cross-channel deconvolution algorithm, both spherical and chromatic aberrations are corrected. Experimental results indicate that using our computational imaging approach, diffractive-refractive optics is an alternative candidate to build light efficient and thin optics for white light imaging.

Optics Express, 2014

Objects imaged through thin scattering media can be reconstructed with the knowledge of the complex transmission function of the diffuser. We demonstrate image reconstruction of static and dynamic objects with numerical phase conjugation in a lensless setup. Data is acquired by single shot intensity capture of an object coherently illuminated and obscured by an inhomogeneous medium, i.e. light diffracted at a specimen is scattered by a polycarbonate diffuser and the resulting speckle field is recorded. As a preparational step, which has to be performed only one time before imaging, the complex speckle field diffracted by the diffuser to the camera chip is measured interferometrically, which allows to reconstruct the transmission function of the diffuser. After insertion of the specimen, the speckle field in the camera plane changes, and the complex field of the sample can be reconstructed from the new intensity distribution. After initial interferometric measurement of the diffuser field, the method is robust with respect to a subsequent misalignment of the diffuser. The method can be extended to image objects placed between a pair of thin scattering plates. Since the object information is contained in a single speckle intensity pattern, it is possible to image dynamic processes at video rate.