Light Fields

This paper proposes a method to extend the depth range of integral imaging systems by representing three-dimensional (3D) images in both virtual-real and real-real forms. The approach combines two integral imaging systems using a transmission-type retroreflector and a beam splitter (BS). The transmission-type retroreflector relays the integrated images from one display to form a real full-color 3D aerial images while preserving the resolution at extended depths. The BS combines the second integral imaging system across various depth ranges, providing additional real or virtual 3D images. The proposed method enhances the applicability of integral imaging systems in mixed-reality environments without requiring complex equipment.

Conventional multifocal displays deliver effective monocular focus cues but lack sufficient motion parallax, while multi-view displays offer motion parallax with no monocular focus cues. To overcome these limitations, we propose a system that leverages the benefit of viewpoint generation and the focus-tunable lens (FTL) to support both motion parallax and accommodation cues in one system. Experimental results demonstrate that the proposed setup successfully delivers full-resolution images with support for both depth cues, providing a more realistic visual experience. A compact design, with experimental results demonstrating its effectiveness, is also introduced to further enhance the system's compactness.

This feature issue of Applied Optics contains a series of selected papers reflecting recent progress of correlation optics and illustrating current trends in vector singular optics, internal energy flows at light fields, optical science of materials, and new biomedical applications of lasers.

"Liquid Sky" is a site-specific sculptural installation, functioning also as a bar, realized for the inauguration of the Onassis Cultural Center (Stegi) in Athens, Greece, in 2010. The work—including all functional elements, bar, and seating arrangements—was conceived, designed, constructed, and supervised in its entirety by visual artist Aemilia Papaphilippou.
This PDF presentation offers a detailed account of the creative and construction process, highlighting the interplay between light, form, and spatial logic. It also includes documentation images and a link to the making-of video.

We present a framework for progressive and interactive rendering with soft shadows and indirect illumination of a triangulated scene. Our method is a multi-pass algorithm that separates the rendering of each main component of radiance in order to update the image as fast as new samples are computed. Those radiance samples are computed at the vertices of multiple recursively subdivided meshes, allowing fast hardware interpolation between the samples. These radiance samples are computed using irradiance values cached in multiple meshes. These meshes separate the direct irradiance from each light source and the indirect one. Using multiple meshes gives us the ability to better reuse samples and to better adapt the sampling density than if a unique mesh was used. We also propose to quickly compute accurate soft shadows and indirect irradiance using the graphics hardware for visibility determination.

Plenoptic cameras or light field cameras are a recent type of imaging devices that are starting to regain some popularity. These cameras are able to acquire the plenoptic function (4D light field) and, consequently, able to output the depth of a scene, by making use of the redundancy created by the multi-view geometry, where a single 3D point is imaged several times. Despite the attention given in the literature to standard plenoptic cameras, like Lytro, due to their simplicity and lower price, we did our work based on results obtained from a multi-focus plenoptic camera (Raytrix, in our case), due to their quality and higher resolution images. In this master thesis, we present an automatic method to estimate the virtual depth of a scene. Since the capture is done using a multi-focus plenoptic camera, we are working with multi-view geometry and lens with different focal lengths, and we can use that to back trace the rays in order to obtain the depth. We start by finding salient poin...

Light field techniques allow the rendering of objects in time complexity unrelated to their geometric complexity. The technique discretely samples the space of light rays exiting the boundary around an object and then reconstructs a requested view from these data. In order to generate high quality images a dense sampling of the space is required which leads to large data sets. These data sets exhibit a high degree of coherence and should be compressed in order to make their size manageable. We present a wavelet-based method for storing light fields over planar domains. The parameterization is based on the Nusselt embedding, which leads to simplifications in shading computations when the light fields are used illumination sources. The wavelet transform exploits the coherence in the data to reduce the size of the data sets by factors of 20 times or more without objectionable deterioration in the rendered images. The wavelet representation also allows a hierarchical representation in which detail can be added incrementally, and in which each coarser view is an appropriately filtered version of the finer detail. The wavelet coefficients are compressed by thresholding the coefficients and storing them in a sparse hexadecary tree. The tree encoding allows random access over the compressed wavelet coefficients which is essential for extracting slices and point samples from the light field.

Light fields are 4D signals capturing rich information from a scene. The availability of multiple views enables scene depth estimation, that can be used to generate 3D point clouds. The constructed 3D point clouds, however, generally contain distortions and artefacts primarily caused by inaccuracies in the depth maps. This paper describes a method for noise removal in 3D point clouds constructed from light fields. While existing methods discard outliers, the proposed approach instead attempts to correct the positions of points, and thus reduce noise without removing any points, by exploiting the consistency among views in a light-field. The proposed 3D point cloud construction and denoising method exploits uncertainty measures on depth values. We also investigate the possible use of the corrected point cloud to improve the quality of the depth maps estimated from the light field.

Light fields are 4D signals capturing rich information from a scene. The availability of multiple views enables scene depth estimation, that can be used to generate 3D point clouds. The constructed 3D point clouds, however, generally contain distortions and artefacts primarily caused by inaccuracies in the depth maps. This paper describes a method for noise removal in 3D point clouds constructed from light fields. While existing methods discard outliers, the proposed approach instead attempts to correct the positions of points, and thus reduce noise without removing any points, by exploiting the consistency among views in a light-field. The proposed 3D point cloud construction and denoising method exploits uncertainty measures on depth values. We also investigate the possible use of the corrected point cloud to improve the quality of the depth maps estimated from the light field.

A single hand-held camera provides an easily accessible but potentially extremely powerful setup for augmented reality. Capabilities which previously required expensive and complicated infrastructure have gradually become possible from a live monocular video feed, such as accurate camera tracking and, most recently, dense 3D scene reconstruction. A new frontier is to work towards recovering the reflectance properties of general surfaces and the lighting configuration in a scene without the need for probes, omnidirectional cameras or specialised light-field cameras. Specular lighting phenomena cause effects in a video stream which can lead current tracking and reconstruction algorithms to fail. However, the potential exists to measure and use these effects to estimate deeper physical details about an environment, enabling advanced scene understanding and more convincing AR. In this paper we present an algorithm for real-time surface lightfield capture from a single hand-held camera, which is able to capture dense illumination information for general specular surfaces. Our system incorporates a guidance mechanism to help the user interactively during capture. We then split the light-field into its diffuse and specular components, and show that the specular component can be used for estimation of an environment map. This enables the convincing placement of an augmentation on a specular surface such as a shiny book, with realistic synthesized shadow, reflection and occlusion of specularities as the viewpoint changes. Our method currently works for planar scenes, but the surface light-field representation makes it ideal for future combination with dense 3D reconstruction methods.

A single hand-held camera provides an easily accessible but potentially extremely powerful setup for augmented reality. Capabilities which previously required expensive and complicated infrastructure have gradually become possible from a live monocular video feed, such as accurate camera tracking and, most recently, dense 3D scene reconstruction. A new frontier is to work towards recovering the reflectance properties of general surfaces and the lighting configuration in a scene without the need for probes, omni-directional ...

This feature issue of Applied Optics contains a series of selected papers reflecting recent progress of correlation optics and illustrating current trends in vector singular optics, internal energy flows at light fields, optical science of materials, and new biomedical applications of lasers.

This paper studies the problem of low rank approximation of light fields for compression. A homography-based approximation method is proposed which jointly searches for homographies to align the different views of the light field together with the low rank approximation matrices. We first consider a global homography per view and show that depending on the variance of the disparity across views, the global homography is not sufficient to well-align the entire images. In a second step, we thus consider multiple homographies, one per region, the region being extracted using depth information. We first show the benefit of the joint optimization of the homographies together with the low-rank approximation. The resulting compact representation is then compressed using HEVC and the results are compared with those obtained by directly applying HEVC on the light field views restructured as a video sequence. The experiments using different data sets show substantial PSNR-rate gain of our method, especially for real light fields.

This paper describes a light field compression scheme based on a novel homography-based low rank approximation method called HLRA. The HLRA method jointly searches for the set of homographies best aligning the light field views and for the low rank approximation matrices. The light field views are aligned using either one global homography or multiple homographies depending on how much the disparity across views varies from one depth plane to the other. The light field low-rank representation is then compressed using HEVC. The best pair of rank and QP parameters of the coding scheme, for a given target bit-rate, is predicted with a model defined as a function of light field disparity and texture features. The results are compared with those obtained by directly applying HEVC on the light field views restructured as a pseudo-video sequence. The experiments using different data sets show substantial PSNR-rate gain of our compression algorithm, as well as the accuracy of the proposed parameter prediction model, especially for real light fields. A scalable extension of the coding scheme is finally proposed.

We introduce, by this work, a fast method to estimate probability density functions in the semi-bounded case. This new technique is a simplified version of the kernel-diffeomorphism estimator which requires complexity in the calculations. It is based on a logarithmic transformation of the data which will be estimated by the conventional kernel estimator. Thus, the algorithm complexity is reduced from O(N 2) to O(N).

We present LCAV-31, a multi-view object recognition dataset designed specifically for benchmarking light field image analysis tasks. The principal distinctive factor of LCAV-31 compared to similar datasets is its design goals and availability of novel visual information for more accurate recognition (i.e. light field information). The dataset is composed of 31 object categories captured from ordinary household objects. We captured the color and light field images using the recently popularized Lytro consumer camera. Different views of each object have been provided as well as various poses and illumination conditions. We explain all the details of different capture parameters and acquisition procedure so that one can easily study the effect of different factors on the performance of algorithms executed on LCAV-31. Moreover, we apply a set of basic object recognition algorithms on LCAV-31. The results of these experiments can be used as a baseline for further development of novel algorithms.

In recent years, we have observed the advent of plenoptic modalities such as light fields, point clouds and holography in many devices and applications. Besides plenty of technical challenges brought by these new modalities, a particular challenge is arising at the horizon, namely providing interoperability between these devices and applications, and-in addition-at a cross-modality level. Based on these observations the JPEG committee (ISO/IEC JTC1/SC29/WG1 and ITU-T SG16) has initiated a new standardization initiative-JPEG Plenothat is intended to define an efficient framework addressing the above interoperability issues. In this paper, an overview is provided about its current status and future plans.

Keyframe extraction process consists on presenting an abstract of the entire video with the most representative frames. It is one of the basic procedures relating to video retrieval and summary. This paper present a novel method for keyframe extraction based on SURF local features. First, we select a group of candidate frames from a video shot using a leap extraction technique. Then, SURF is used to detect and describe local features on the candidate frames. After that, we analyzed those features to eliminate near duplicate keyframes, helping to keep a compact set, using FLANN method. We developed a comparative study to evaluate our method with three state of the art approaches based on local features. The results show that our method overcomes those approaches

We introduce the concept of vision-realistic rendering-the computer generation of synthetic images that incorporate the characteristics of a particular individual's entire optical system. Specifically, this paper develops a method for simulating the scanned foveal image from wavefront data of actual human subjects, and demonstrates those methods on sample images. First, a subject's optical system is measured by a Shack-Hartmann wavefront aberrometry device. This device outputs a measured wavefront which is sampled to calculate an object space point spread function (OSPSF). The OSPSF is then used to blur input images. This blurring is accomplished by creating a set of depth images, convolving them with the OSPSF, and finally compositing to form a vision-realistic rendered image. Although processing in image space allows an increase in speed, the images may have artifacts introduced due to occlusion or discretization. Two approaches for object identification to properly blur the scene are discussed. Applications of vision-realistic rendering in computer graphics as well as in optometry and ophthalmology are discussed. keywords

MIT Media Lab 2 NVIDIA Research 3 front layer watch face corrected for presbyopia conventional display single-layer pre-filtering two-layer pre-filtering perceived images displayed images Figure 1: Correcting presbyopia using multilayer displays. A presbyopic individual observes a watch at a distance of 45 cm. The watch appears out of focus due to the limited range of accommodation. To read the watch, corrective eyewear (e.g., bifocals) must be worn with a +2.50 diopter spherical lens. (Left) As a substitute for eyewear, the watch can be modified to use a multilayer display containing two semi-transparent, light-emitting panels. The images displayed on these layers are pre-filtered such that the watch face appears in focus when viewed by the defocused eye. (Right) From left to right along the top row: the perceived image using a conventional display (e.g., an unmodified LCD), using prior single-layer pre-filtering methods, and using the proposed multilayer pre-filtering method. Corresponding images of the watch face are shown along the bottom row. Two-layer pre-filtering, while increasing the watch thickness by 6 mm, enhances contrast and eliminates ringing artifacts, as compared to prior single-layer pre-filtering methods. Watch image c Timex Group USA, Inc.

Conventional Display Multilayer Display high-res, low contrast Light Field Display-direct low-res, high contrast Proposed Display high-res, high contrast single display panel (no processing) [Huang et al. 2012] (prefiltering) [Pamplona et al. 2012] (no processing) (prefiltering)

We introduce the concept of vision-realistic rendering-the generation of images that incorporate characteristics of a particular individual's optical system. We then describe a pipeline for creating vision-realistic images. First, a subject's optical system is measured by a Shack-Hartmann wavefront aberrometry device. This device outputs a measured wavefront which is sampled to calculate an object space point spread function (OSPSF). The OSPSF is then used to blur input images. This blurring is accomplished by creating a set of depth images, convolving them with the OSPSF, and finally compositing to form a vision-realistic image. We discuss applications of visionrealistic rendering in computer graphics as well as in optometry and ophthalmology and note that our method is a post-process and can handle simple camera models as a special case.

Given the measurements of the optical aberrations of a user's eye, a vision correcting display will present a transformed image that when viewed by this individual will appear in sharp focus. This could impact computer monitors, laptops, tablets, and mobile phones. Vision correction could be provided in some cases where spectacles are ineffective.

Vision-realistic rendering is the computer generation of synthetic images that incorporate the characteristics of a particular individual's entire optical system. Specifically, we simulate the scanned foveal image from wavefront data of actual human subjects. The concept of a vision correcting display involves digitally modifying the content of a display using measurements of the optical aberrations of the viewer's eye so that the display can be seen in sharp focus by the user without requiring the use of eyeglasses or contact lenses. Given the measurements of the optical aberrations of a user's eye, a vision correcting display will present a transformed image that when viewed by this individual will appear in sharp focus. Vision correction could be provided in some cases where spectacles are ineffective.

Figure 1: Two implementations of the Frankencamera architecture: (a) The custom-built F2-portable and self-powered, best for projects requiring flexible hardware. (b) A Nokia N900 with a modified software stack-a compact commodity platform best for rapid development and deployment of applications to a large audience.

The real-time development of multi-camera systems is a great challenge. Synchronization and large data rates of the cameras adds to the complexity of these systems as well. The complexity of such system also increases as the number of their incorporating cameras increases. The customary approach to implementation of such system is a central type, where all the raw stream from the camera are first stored then processed for their target application. An alternative approach is to embed smart cameras to these systems instead of ordinary cameras with limited or no processing capability. Smart cameras with intra and inter camera processing capability and programmability at the software and hardware level will offer the right platform for distributed and parallel processing for multicamera systems real-time application development. Inter camera processing requires the interconnection of smart cameras in a network arrangement. A novel hardware emulating platform is introduced for demonstrating the concept of the interconnected network of cameras. A methodology is demonstrated for the interconnection network of camera construction and analysis. A sample application is developed and demonstrated.

The Frankencamera (FCam) architecture and API enables precise control over the camera in computational photography applications. We present an extension to FCam API for systems equipped with multiple cameras. The proposed extension allows for an enumeration of cameras and their corresponding properties, such as position or orientation. In addition, we explicitly support camera synchronization, either through hardware mechanisms or software primitives. If hardware synchronization is available, cameras can be grouped together under a concept of a multi-sensor. Otherwise, multiple camera streams are scheduled asynchronously and synchronized using our software control primitives.

Bright-field microscopy of sputum samples is still the most widely used Tuberculosis (TB) diagnostic method in countries facing a high incidence of TB. However, this diagnostic method, because it is a visual analysis, requires attention and training of those who perform it, and presents high intra-and interobserved variation. As a result, many research groups are working on methods of automatic detection of bacilli, aiming to automate this process. The fact that not all of the bacilli present in the examined microscope field are in focus increases the challenge faced by researchers in obtaining automatic methods of detecting bacilli. Fusion images can be a means of overcoming this problem, combining multiple images, from the same field, with diverse focuses into a single focused one. In this paper, we present a multi-focus image fusion method applied to conventional sputum smear microscopy images. The goal is to establish the best method to obtain an extended focus microscopy image where all bacilli present in the field are in focus. The proposed method was compared with three other techniques from the literature by using Variance and Multichannel Q AB/F metrics. The proposed method exhibited the best balance of quality evidence (focus and preservation of information).

Skin cancer is one of the most dangerous types of cancers that affect millions of people every year. The detection of skin cancer in the early stages is an expensive and challenging process. In recent studies, machine learning-based methods help dermatologists in classifying medical images. This paper proposes a deep learning-based model to detect and classify skin cancer using the concept of deep Convolution Neural Network (CNN). Initially, we collected a dataset that includes four skin cancer image data before applying them in augmentation techniques to increase the accumulated dataset size. Then, we designed a deep CNN model to train our dataset. On the test data, our model receives 95.98% accuracy that exceeds the two pre-train models, GoogleNet by 1.76% and MobileNet by 1.12%, respectively. The proposed deep CNN model also beats other contemporaneous models while being computationally comparable.

Patch-based image synthesis methods have been successfully applied for various editing tasks on still images, videos and stereo pairs. In this work we extend patch-based synthesis to plenoptic images captured by consumer-level lenselet-based devices for interactive, efficient light field editing. In our method the light field is represented as a set of images captured from different viewpoints. We decompose the central view into different depth layers, and present it to the user for specifying the editing goals. Given an editing task, our method performs patch-based image synthesis on all affected layers of the central view, and then propagates the edits to all other views. Interaction is done through a conventional 2D image editing user interface that is familiar to novice users. Our method correctly handles object boundary occlusion with semi-transparency, thus can generate more realistic results than previous methods. We demonstrate compelling results on a wide range of applicati...

This paper proposes a fast, stable and accurate meshless method to simulate geometrically non-linear elastic behaviors. To address the inherent limitations of finite element (FE) models, the discretization of the domain is simplified by removing the need to create polyhedral elements. The volumetric locking effect exhibited by incompressible materials in some linear FE models is also completely avoided. Our approach merely requires that the volume of the object be filled with a cloud of points. To minimize numerical errors, we construct a corotational formulation around the quadrature positions that is well suited for large displacements containing small deformations. The equations of motion are integrated in time following an implicit scheme. The convergence rate and accuracy are validated through both stretching and bending case studies. Finally, results are presented using a set of examples that show how we can easily build a realistic physical model of various deformable bodies with little effort spent on the discretization of the domain.

This paper addresses the problem of using deep learning for makeup style transfer. For solving this problem, we propose a new supervised method. Additionally, we present a technique for creating a synthetic dataset for makeup transfer used to train our model. The obtained results were compared with six popular methods for makeup transfer using three metrics. The tests were carried out on four available data sets. The proposed method, in many respects, is competitive with the methods used in the literature. Thanks to images of faces with generated synthetic makeup, the proposed method learns to better transfer details, and the learning process is significantly accelerated.

Light field imaging is characterized by capturing brightness, color, and directional information of light rays in a scene. This leads to image representations with huge amount of data that require efficient coding schemes. In this paper, lenslet images are rendered into sub-aperture images. These images are organized as a pseudo-sequence input for the HEVC video codec. To better exploit redundancy among the neighboring sub-aperture images and consequently decrease the distances between a sub-aperture image and its references used for prediction, sub-aperture images are divided into four smaller groups that are scanned in a serpentine order. The most central sub-aperture image, which has the highest similarity to all the other images, is used as the initial reference image for each of the four regions. Furthermore, a structure is defined that selects spatially adjacent sub-aperture images as prediction references with the highest similarity to the current image. In this way, encoding...

Efficient compression plays a significant role in Light Field imaging technology because of the huge amount of data needed for their representation. Video encoders using different strategies are commonly used for Light Field image compression. In this paper, different video encoder implementations including HM, VTM, x265, xvc, VP9, and AV1 are analysed and compared in terms of coding efficiency, and encoder/decoder time-complexity. Light field images are compressed as pseudo-videos.

Recent development of computer-assisted medical systems, based on statistical shape analysis, leads to a growing number of emerging shape and appearance models. In this paper, we propose a novel method for a lossy compression of 3D statistical shape and intensity models exploiting the JPEG 2000 image coding system. We also investigate the influence of the lossy compression on the accuracy of an atlas-based 2D/3D reconstruction, which is one of the common applications of the statistical appearance models. The results revealed the method is highly effective for the intensity information, reaching thousandfold compression ratios without affecting the 2D/3D reconstruction accuracy. The bitrate of shape information can be compressed several times without significant influence on the reconstruction accuracy.

Light field data records the amount of light at multiple points in space, captured e.g. by an array of cameras or by a light-field camera that uses microlenses. Since the storage and transmission requirements for such data are tremendous, compression techniques for light fields are gaining momentum in recent years. Although plenty of efficient compression formats do exist for still and moving images, only a little research on the impact of these methods on light field imagery is performed. In this paper, we evaluate the impact of state-of-the-art image and video compression methods on quality of images rendered from light field data. The methods include recent video compression standards, especially AV1 and XVC finalised in 2018. To fully exploit the potential of common image compression methods on four-dimensional light field imagery, we have extended these methods into three and four dimensions. In this paper, we show that the four-dimensional light field data can be compressed much more than independent still images while maintaining the same visual quality of a perceived picture. We gradually compare the compression performance of all image and video compression methods, and eventually answer the question, "What is the best compression method for light field data?".

Image compression play significant role in the data transfer and storage. Recently Machine learning has achieved tremendous success in various domain of image processing. Further due to large increase in image data, the research activities in image compression continue to preserve bandwidth or storage resources. This paper focuses on research on the combination of traditional compression algorithms and machine learning techniques. Both compression and learning algorithms are discussed, providing background and reasoning for their combination. This help to understand the current trends and future scope in image compression using machine learning.

In the presence of matter there is no fundamental limit preventing confinement of visible light even down to atomic scales. Achieving such confinement and the corresponding intensity enhancement inevitably requires simultaneous control over atomic-scale details of material structures and over the optical modes that such structures support. By means of self-assembly we have obtained side-by-side aligned gold nanorod dimers with robust atomically-defined gaps reaching below 0.5 nm. The existence of atomically-confined light fields in these gaps is demonstrated by observing extreme Coulomb splitting of corresponding symmetric and antisymmetric dimer eigenmodes of more than 800 meV in white-light scattering experiments. Our results open new perspectives for atomically-resolved spectroscopic imaging, deeply nonlinear optics, ultra-sensing, cavity optomechanics as well as for the realization of novel quantum-optical devices.

The paper describes a method for measuring the similarity and symmetry of an image annotated with bounding boxes indicating image objects. The latter representation became popular recently due to the rapid development of fast and efficient deep-learning-based object-detection methods. The proposed approach allows for comparing sets of bounding boxes to estimate the degree of similarity of their underlying images. It is based on the fuzzy approach that uses the fuzzy mutual position (FMP) matrix to describe spatial composition and relations between bounding boxes within an image. A method of computing the similarity of two images described by their FMP matrices is proposed and the algorithm of its computation. It outputs the single scalar value describing the degree of contentbased image similarity. By modifying the method's parameters, instead of similarity, the reflectional symmetry of object composition may also be measured. The proposed approach allows for measuring differences in objects' composition of various intensities. It is also invariant to translation and scaling and-in case of symmetry detection-position and orientation of the symmetry axis. A couple of examples illustrate the method.

Most virtual globe systems feature a rendering of the atmosphere that surrounds the earth. This element is so widespread that seeing a virtual earth without a surrounding halo makes the image seem much more artificial. Atmospheric rendering is a costly process aimed to enhance the realism and beauty of the scene. For that reason many virtual globe systems, specially in low-resourced devices, rely on simplified schemes to represent the atmosphere. However, the accurate representation of the atmosphere implies the volumetric rendering of the semi-transparent air mass that covers the whole geographical scene. Thus, the color of each pixel is composed of the light scattered by all the points in space projected on that pixel. The present work takes advantage of the spherical symmetry of the atmosphere's mathematical model to implement efficiently this volumetric rendering on mobile devices.

A shallow depth-of-field image keeps the subject in focus, and the foreground and background contexts blurred. This effect requires much larger lens apertures than those of smartphone cameras. Conventional methods acquire RGB-D images and blur image regions based on their depth. However, this approach is not suitable for reflective or transparent surfaces, or finely detailed object silhouettes, where the depth value is inaccurate or ambiguous. We present a learning-based method to synthesize the defocus blur in shallow depth-of-field images from handheld bursts acquired with a single small aperture lens. Our deep learning model directly produces the shallow depth-of-field image, avoiding explicit depth-based blurring. The simulated aperture diameter equals the camera translation during burst acquisition. Our method does not suffer from artifacts due to inaccurate or ambiguous depth estimation, and it is wellsuited to portrait photography.

A shallow depth-of-field image keeps the subject in focus, and the foreground and background contexts blurred. This effect requires much larger lens apertures than those of smartphone cameras. Conventional methods acquire RGB-D images and blur image regions based on their depth. However, this approach is not suitable for reflective or transparent surfaces, or finely detailed object silhouettes, where the depth value is inaccurate or ambiguous. We present a learning-based method to synthesize the defocus blur in shallow depth-of-field images from handheld bursts acquired with a single small aperture lens. Our deep learning model directly produces the shallow depth-of-field image, avoiding explicit depth-based blurring. The simulated aperture diameter equals the camera translation during burst acquisition. Our method does not suffer from artifacts due to inaccurate or ambiguous depth estimation, and it is wellsuited to portrait photography.

The goal of this research is to provide linguists with visualisations for analysing the results of their hate speech annotation. These visualisations consist of a set of interactive graphs for analysing the global distribution of annotated messages, finding relationships between features, and detecting inconsistencies in the annotation. We used a corpus that includes 1,262 comments posted in response to different Spanish online new articles. The comments were annotated with features such as sarcasm, mockery, insult, improper language, constructivity and argumentation, as well as with level of toxicity ('not-toxic', 'mildly toxic', 'toxic' or 'very toxic'). We evaluated the selected visualisations with users to assess the graphs' comprehensibility, interpretability and attractiveness. One of the lessons learned from the study is the usefulness of mixed visualisations that include simple graphs (Bar, Heat map)-to facilitate the familiarisation with the results of the annotated corpus together with more complex ones (Sankey, Spider or Chord)-to explore and identify relationships between features and to find inconsistencies.

With the ongoing improvement of digital cameras and smartphones, more and more people can acquire high- resolution digital images. Due to their size and high performance requirements, such Gigapixel Images (GPIs) are often challenging to process and explore compared to conventional low resolution images. To address this problem, this paper presents a service-based approach for GPI processing in a device-independent way using cloud-based processing. For it, the concept, design, and implementation of GPI processing functionality into service-based architectures is presented and evaluated with respect to advantages, limitations, and runtime performance.

Recent realizations of hand-held plenoptic cameras have given rise to previously unexplored effects in photography. Designing a mobile phone plenoptic camera is becoming feasible with the significant increase of computing power of mobile devices and the introduction of System on a Chip. However, capturing high numbers of views is still impractical due to special requirements such as ultra-thin camera and low costs. In this paper, we analyze a mobile plenoptic camera solution with a small number of views. Such a camera can produce a refocusable high resolution final image if a depth map is generated for every pixel in the sparse set of views. With the captured multi-view images, the obstacle to recovering a high-resolution depth is occlusions. To robustly resolve these, we first analyze the behavior of pixels in such situations. We show that even under severe occlusion, one can still distinguish different depth layers based on statistics. We estimate the depth of each pixel by discretizing the space in the scene and conducting plane sweeping. Specifically, for each given depth, we gather all corresponding pixels from other views and model the in-focus pixels as a Gaussian distribution. We show how it is possible to distinguish occlusion pixels, and in-focus pixels in order to find the depths. Final depth maps are computed in real scenes captured by a mobile plenoptic camera.

Dynamic light fields provide a richer, more realistic 3D representation of a moving scene. However, this leads to higher data rates since excess storage and transmission requirements are needed. We propose a novel approach to efficiently represent and encode dynamic light field data for display applications based on dynamic mode decomposition (DMD). Acquired images are firstly obtained through optimized coded aperture patterns for each temporal frame/camera viewpoint of a dynamic light field. The underlying spatial, angular, and temporal correlations are effectively exploited by a data-driven DMD on these acquired images arranged as time snapshots. Next, High Efficiency Video Coding (HEVC) removes redundancies in light field data, including intra-frame and inter-frame redundancies, while maintaining high reconstruction quality. The proposed scheme is the first of its kind to treat light field videos as mathematical dynamical systems, leverage on dynamic modes of acquired images, and gain flexible coding at various bitrates. Experimental results demonstrate our scheme's superior compression efficiency and bitrate savings compared to the direct encoding of acquired images using HEVC codec.