A Morphable Face Albedo Model

The colours of objects perceived by a colour camera are dependent on the illumination conditions. For example, when the prevailing illumination condition does not correspond to the one used in the white balancing of the camera, the object colours can change their appearance due to the lack of colour constancy capabilities. Many methods for colour constancy have been suggested but so far their performance has been inadequate. Faces are common and important objects encountered in many applications. Therefore, this thesis is dedicated to studying face colours and their robust use under real world illumination conditions. The main thesis statement is "knowledge about an object's colour, like skin colour changes under different illumination conditions, can be used to develop more robust techniques against illumination changes". Many face databases exist, and in some cases they contain colour images and even videos. However, from the point of view of this thesis these databases have several limitations: unavailability of spectral data related to image acquisition, undefined illumination conditions of the acquisition, and if illumination change is present it often means only change in illumination direction. To overcome these limitations, two databases, a Physics-Based Face Database and a Face Video Database were created. In addition to the images, the Physics-Based Face Database consists of spectral data part including skin reflectances, channel responsivities of the camera and spectral power distribution of the illumination. The images of faces are taken under four known light sources with different white balancing illumination conditions for over 100 persons. In addition to videos, the Face Video Database has spectral reflectances of skin for selected persons and images taken with the same measurement arrangement as in the Physics-Based Face Database. The images and videos are taken with several cameras. The databases were used to gather information about skin chromaticities and to provide test material. The skin RGB from images were converted to different colour spaces and the result showed that the normalized colour coordinate was among the most usable colour spaces for skin chromaticity modelling. None of the colour spaces could eliminate the colour shifts in chromaticity. The obtained chromaticity constraint can be implemented as an adaptive skin colour modelling part of face tracking algorithms, like histogram backprojection or mean shift. The performances of these adaptive algorithms were superior compared to those using a fixed skin colour model or model adaptation based on spatial pixel selection. Of course, there are cases when the colour cue is not enough alone and use of other cues like motion or edge data would improve the result. It was also demonstrated that the skin colour model can be used to segment faces and the segmentation results depend on the background due to the method used. Also an application for colour correction using principal component analysis and a simplified dichromatic reflection model was shown to improve colour quality of seriously clipped images. The results of tracking, segmentation and colour correction experiments using the collected data validate the thesis statement.

2007

In this paper, we present a new method to change the illumination condition of a face image, with unknown face geometry and albedo information. This problem is particularly difficult when there is only one single image of the subject available and it was taken under a harsh lighting condition. Recent research demonstrates that the set of images of a convex Lambertian object obtained under a wide variety of lighting conditions can be approximated accurately by a low-dimensional linear subspace using spherical harmonic representation. However, the approximation error can be large under harsh lighting conditions [2] thus making it difficult to recover albedo information. In order to address this problem, we propose a subregion based framework that uses a Markov Random Field to model the statistical distribution and spatial coherence of face texture, which makes our approach not only robust to harsh lighting conditions, but insensitive to partial occlusions as well. The performance of our framework is demonstrated through various experimental results, including the improvement to the face recognition rate under harsh lighting conditions.

2008 8th IEEE International Conference on Automatic Face & Gesture Recognition, 2008

We propose a novel method to correct for arbitrary illumination variation in the face images. The main purpose is to improve recognition results of face images taken under uncontrolled illumination conditions. We correct the illumination variation in the face images using a face shape model, which allows us to estimate the face shape in the face image. Using this face shape, we can reconstruct a face image under frontal illumination. These reconstructed images improve the results in face identification. We experimented both with face images acquired under different controlled illumination conditions in a laboratory and under uncontrolled illumination conditions.

IEEE Access, 2020

In this paper, we present a novel facial albedo and 3D shape recovery method with a local spherical harmonic illumination model. From a face in an image, the proposed method can produce a highquality 3D shape and albedo using a novel parameterization of local illuminations. Because a facial shape is partially convex, a single spherical harmonics is generally used for the illumination of a face within a constrained illumination environment. However, when a facial image is captured in an unconstrained scene, the illumination is inappropriately estimated due to the presence of shadow and specular reflections. To address this issue, we propose a novel local spherical harmonic illumination model for representing the illumination of a face. Unlike the existing parameterization of local illumination, our local spherical harmonic illumination model utilizes a smooth weight function for the seamless representation of natural illumination. Therefore, the albedo and shape information in an image can be precisely estimated using the first-order spherical harmonics. For accurate estimation of albedo, we also utilize facial albedo statistics to prevent the estimated albedo from becoming biased toward input image. Furthermore, we developed an accurate and reliable 3D shape reconstruction method from a normal map based on tetrahedron-based deformation. Comparing to the Laplacian deformation based method, our method is applicable to any mesh regardless of its structure. Through rigorous experiments, we demonstrate that the proposed local spherical harmonic illumination model is effective in estimating the complex illumination and can recover a high-quality facial albedo and 3D shape.

Computer Vision/ …, 2009

We present a fast and robust iterative method for interpreting face images under non-uniform lighting conditions by using a fitting algorithm which utilizes an illumination-based 3D active appearance model in order to fit a face model to an input face image. Our method is based on improving the Jacobian each iteration using the parameters of lighting that have been estimated in preceding iterations. In the training stage, we precalculate a set of synthetic face images of basis reflectances and albedo generated from displacing one at the time, each one of the model parameters, and subsequently, in the fitting stage, we use all these images in combination with lighting parameters for assembling a Jacobian matrix adapted to the illumination estimated in the last iteration. In contrast to other works where an initial pose is required to begin the fit, our approach only uses a simple initialization in translation and scale. At the end of the fitting process, our algorithm obtains a compact set of parameters of albedo, 3D shape, 3D pose and illumination which describe the appearance of the input face image.

arXiv (Cornell University), 2022

Image enhancement helps to generate balanced lighting distributions over faces. Our goal is to get an illuminancebalanced enhanced face image from a single view. Traditionally, image enhancement methods ignore the 3D geometry of the face or require a complicated multi-view geometry. Other methods cause color tone shifting or over saturation. Inspired by the new research achievements in face alignment and face 3D modeling, we propose an improved face image enhancement method by leveraging 3D face models. Given a face image as input, our method will first estimate its lighting distribution. Then we build an optimization process to refine the distribution. Finally, we generate an illuminance-balanced face image from a single view. Experiments on the FiveK dataset [5] demonstrate that our method performs well and compares favorably with other methods.

2005

3D Face modeling is still one of the biggest challenges in computer graphics. In this paper we present a novel framework that acquires the 3D shape, texture, pose and illumination of a face from a single photograph. Additionally, we show how we can recreate a face under varying illumination conditions. Or, essentially relight it. Using a custom-built face scanning system, we have collected 3D face scans and light reflection images of a large and diverse group of human subjects. We derive a morphable face model for 3D face shapes and accompanying textures by transforming the data into a linear vector sub-space. The acquired images of faces under variable illumination are then used to derive a bilinear illumination model that spans 3D face shape and illumination variations. Using both models we, in turn, propose a novel fitting framework that estimates the parameters of the morphable model given a single photograph. Our framework can deal with complex face reflectance and lighting environments in an efficient and robust manner. In the results section of our paper, we compare our methods to existing ones and demonstrate its efficacy in reconstructing 3D face models when provided with a single photograph. We also provide several examples of facial relighting (on 2D images) by performing adequate decomposition of the estimated illumination using our framework.

2017

IEEE Transactions on Pattern Analysis and Machine Intelligence

We propose HyFRIS-Net to jointly estimate the hybrid reflectance and illumination models, as well as the refined face shape from a single unconstrained face image in a pre-defined texture space. The proposed hybrid reflectance and illumination representation ensure photometric face appearance modeling in both parametric and non-parametric spaces for efficient learning. While forcing the reflectance consistency constraint for the same person and face identity constraint for different persons, our approach recovers an occlusion-free face albedo with disambiguated color from the illumination color. Our network is trained in a self-evolving manner to achieve general applicability on real-world data. We conduct comprehensive qualitative and quantitative evaluations with state-of-the-art methods to demonstrate the advantages of HyFRIS-Net in modeling photo-realistic face albedo, illumination, and shape.

2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022

Most face relighting methods are able to handle diffuse shadows, but struggle to handle hard shadows, such as those cast by the nose. Methods that propose techniques for handling hard shadows often do not produce geometrically consistent shadows since they do not directly leverage the estimated face geometry while synthesizing them. We propose a novel differentiable algorithm for synthesizing hard shadows based on ray tracing, which we incorporate into training our face relighting model. Our proposed algorithm directly utilizes the estimated face geometry to synthesize geometrically consistent hard shadows. We demonstrate through quantitative and qualitative experiments on Multi-PIE and FFHQ that our method produces more geometrically consistent shadows than previous face relighting methods while also achieving state-of-the-art face relighting performance under directional lighting. In addition, we demonstrate that our differentiable hard shadow modeling improves the quality of the estimated face geometry over diffuse shading models.