Academia.eduAcademia.edu

Outline

Title
Abstract
Key Takeaways
Figures
Introduction
Related Work
Results
Conclusions
References

Relighting Character Motion for Photoreal Simulations

Profile image of Paul DebevecPaul Debevec

2006

Abstract

Abstract: We present a fully image-based approach for capturing and modeling real human locomotion under varying illumination and viewpoint that overviews the techniques and results presented by [Einarsson et al, 2006]. An actor performs repeatable locomotive actions (walking/running) on a rotating treadmill while being filmed from a vertical array of 3 high-speed cameras under controlled rapidly changing lighting conditions.

Key takeaways
sparkles

AI

  1. This work achieves image-based relighting and viewpoint control for realistic human locomotion simulations.
  2. The method uses a rotating treadmill and three high-speed cameras to capture motion under varied lighting.
  3. Optical flow is employed to correct perturbations and align images temporally, yielding a flowed reflectance field.
  4. The system processes 3 × 8GB of raw data into approximately 1.5GB of reflectance fields and 1GB of flow maps.
  5. Realistic composites integrate subjects into both real and virtual environments, enhancing training simulations.
Figures (10)
Figure 1. Multiple instances of a captured subject rendered into an image-based lighting environment. Bruce Lamond*, Charles-Felix Chabert, Per Einarsson, Andrew Jones, Wan-Chun Ma, Tim Hawkins , Mark Bolas* Sebastian Sylwan, Paul Debevec University of Southem California Institute of Creative Technologies, LA, CA, 90292 ‘University of Southern California Cinema-Television Interactive Media Division, LA, CA 90089
Figure 1. Multiple instances of a captured subject rendered into an image-based lighting environment. Bruce Lamond*, Charles-Felix Chabert, Per Einarsson, Andrew Jones, Wan-Chun Ma, Tim Hawkins , Mark Bolas* Sebastian Sylwan, Paul Debevec University of Southem California Institute of Creative Technologies, LA, CA, 90292 ‘University of Southern California Cinema-Television Interactive Media Division, LA, CA 90089
Public reporting burden for the collection of information is estimated to average 1 hour per response, including the time for reviewing instructions, searching existing data sources, gathering and maintaining the data needed, and completing and reviewing the collection of information. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden, to Washington Headquarters Services, Directorate for Information Operations and Reports, 1215 Jefferson Davis Highway, Suite 1204, Arlington VA 22202-4302. Respondents should be aware that notwithstanding any other provision of law, no person shall be subject to a penalty for failing to comply with a collection of information if it does not display a currently valid OMB control number.
Public reporting burden for the collection of information is estimated to average 1 hour per response, including the time for reviewing instructions, searching existing data sources, gathering and maintaining the data needed, and completing and reviewing the collection of information. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden, to Washington Headquarters Services, Directorate for Information Operations and Reports, 1215 Jefferson Davis Highway, Suite 1204, Arlington VA 22202-4302. Respondents should be aware that notwithstanding any other provision of law, no person shall be subject to a penalty for failing to comply with a collection of information if it does not display a currently valid OMB control number.
Figure 2. A side view schematic of our capture stage.
Figure 2. A side view schematic of our capture stage.
Figure 3. (a) 1 set of basis lighting conditions taken from the middle camera in 1/30" sec. The sequence shows 26 basis lighting conditions, 3 matte and track frames (and 1 unused stripe pattern). (b)The 36x3 array of viewpoints for a single pose in the cycle.
Figure 3. (a) 1 set of basis lighting conditions taken from the middle camera in 1/30" sec. The sequence shows 26 basis lighting conditions, 3 matte and track frames (and 1 unused stripe pattern). (b)The 36x3 array of viewpoints for a single pose in the cycle.
An alpha channel [Porter and Duff, 1984] or matte is generated for each tracking frame To, T;, and T, in each lighting set. The alpha channel is derived from the track frame T and the matte frame M following each track frame. During a matte lighting condition, the main lights for the other lighting directions are tumed off and the special matting lights are tured on to light the retro- reflective cloth and matting backdrop. To compute the matte we compare neighboring track and matte frames and retain inferred foreground pixels from the track frame whose monochrome brightnesses are greater then in the matte frame. We then eliminate stray foreground elements from the track frame by excluding pixels not part of the argest central connected matte component and apply a 1- pixel Gaussian blur to model the filtering introduced by the image sensor and color interpolation processes to yield the final matte a. The pre-multiplied foreground T’ is computed by matting T onto a black background using the clean plate image C and a using T’ =T - (1 - a).C Fig. 5). Figure 5. Matting. A tracking frame T, consecutive matte frame M, alpha matte image a, and T matted onto black and color-corrected T’
An alpha channel [Porter and Duff, 1984] or matte is generated for each tracking frame To, T;, and T, in each lighting set. The alpha channel is derived from the track frame T and the matte frame M following each track frame. During a matte lighting condition, the main lights for the other lighting directions are tumed off and the special matting lights are tured on to light the retro- reflective cloth and matting backdrop. To compute the matte we compare neighboring track and matte frames and retain inferred foreground pixels from the track frame whose monochrome brightnesses are greater then in the matte frame. We then eliminate stray foreground elements from the track frame by excluding pixels not part of the argest central connected matte component and apply a 1- pixel Gaussian blur to model the filtering introduced by the image sensor and color interpolation processes to yield the final matte a. The pre-multiplied foreground T’ is computed by matting T onto a black background using the clean plate image C and a using T’ =T - (1 - a).C Fig. 5). Figure 5. Matting. A tracking frame T, consecutive matte frame M, alpha matte image a, and T matted onto black and color-corrected T’
Our shadow computation technique models a first order approximation of how the subject should interact with the ground plane to add a qualitative sense of realism to the composite. To compute the shadows we model a 3m x 3m plane below the subject pixels. Volumetric intersection of sp defined from a number of mattes o consisting of 1287 ace [Szeliski, 1993] f a pose give us an approximate visual hull of the subject. Rays are traced from each ground pixel towards tl he light sources to determine the percentage of ligh t from each basis condition that remains visible. The result is a set of 23 attenuation maps (one for each lighting basis in the upper hemisphere) for each pose in the locomotion (Fig. 7). ‘igure 7. Shadow maps for one frame in a walk cycle for the 23 basis lighting conditions above ground.
Our shadow computation technique models a first order approximation of how the subject should interact with the ground plane to add a qualitative sense of realism to the composite. To compute the shadows we model a 3m x 3m plane below the subject pixels. Volumetric intersection of sp defined from a number of mattes o consisting of 1287 ace [Szeliski, 1993] f a pose give us an approximate visual hull of the subject. Rays are traced from each ground pixel towards tl he light sources to determine the percentage of ligh t from each basis condition that remains visible. The result is a set of 23 attenuation maps (one for each lighting basis in the upper hemisphere) for each pose in the locomotion (Fig. 7). ‘igure 7. Shadow maps for one frame in a walk cycle for the 23 basis lighting conditions above ground.
Figure 6. Bi-directional optical flow maps for neighboring viewpoints in the dataset. Up/down displacement is green and left/right is red.
Figure 6. Bi-directional optical flow maps for neighboring viewpoints in the dataset. Up/down displacement is green and left/right is red.
We have implemented flowed light field interpolation on the GPU using OpenGL. The 36 x 3 RGBA images and 180 flow maps for a single pose can be held in 256MB of GPU memory and rendered interactively. To render an animated sequence the images and flow maps for each virtual camera are sent to the GPU. The polygonal geometry of the cylinder is then computed for each pixel. We then pass the texture coordinates and ray intersection point to the fragment shader where virtual camera parameters are inferred along with interpolation coefficients used to warp and blend the contribution from the 4 closest input images. Figure 8. Novel view generation in a flowed light field 6.3 Shadowing and Compositing.
We have implemented flowed light field interpolation on the GPU using OpenGL. The 36 x 3 RGBA images and 180 flow maps for a single pose can be held in 256MB of GPU memory and rendered interactively. To render an animated sequence the images and flow maps for each virtual camera are sent to the GPU. The polygonal geometry of the cylinder is then computed for each pixel. We then pass the texture coordinates and ray intersection point to the fragment shader where virtual camera parameters are inferred along with interpolation coefficients used to warp and blend the contribution from the 4 closest input images. Figure 8. Novel view generation in a flowed light field 6.3 Shadowing and Compositing.
Figure 9. Results. (a-c) walking male subject composited into image-based ‘Uffizi’ environment. (d-f) female subject added to virtual scene with global illumination effects. (g-i) running male composited into a different image-based lighting environment.
Figure 9. Results. (a-c) walking male subject composited into image-based ‘Uffizi’ environment. (d-f) female subject added to virtual scene with global illumination effects. (g-i) running male composited into a different image-based lighting environment.

Related papers

Postproduction re-illumination of live action using time-multiplexed lighting
Paul Debevec

2004

Abstract: In this work, we present a technique for capturing a time-varying human performance in such a way that it can be re-illuminated in postproduction. The key idea is to illuminate the subject with a variety of rapidly changing time-multiplexed basis lighting conditions, and to record these lighting conditions with a fast enough video camera so that several or many different basis lighting conditions are recorded during the span of the final video's desired frame rate.

View PDFchevron_right
A lighting reproduction approach to live-action compositing
Paul Debevec

2002

Abstract We describe a process for compositing a live performance of an actor into a virtual set wherein the actor is consistently illuminated by the virtual environment. The Light Stage used in this work is a two-meter sphere of inward-pointing RGB light emitting diodes focused on the actor, where each light can be set to an arbitrary color and intensity to replicate a real-world or virtual lighting environment.

View PDFchevron_right
The Light Stages and Their Applications to Photoreal Digital Actors
Paul Debevec

Abstract The Light Stage systems built at UC Berkeley and USC ICT have enabled a variety of facial scanning and reflectance measurement techniques that have been explored in several research papers and used in various commercial applications. This short paper presents the evolutionary history of the Light Stage Systems and some of the techniques and applications they have enabled. Keywords: 3D scanning, facial animation, computational illumination

View PDFchevron_right
A global human walking model with real-time kinematic personification
Nadia Magnenat-halmann, Ronan Boulic

The Visual Computer, 1990

This paper presents a human walking model built from experimental data based on a wide range of normalized velocities. The model is structured on two levels. On the first level, global spatial and temporal characteristics (normalized length and step duration) are generated. On the second level, a set of parameterized trajectories produce both the position of the body in space and the internal body configuration. This is performed for a standard structure and an average configuration of the human body. The experimental context corresponding to the model is extended by allowing a continuous variation of global spatial and temporal parameters according to the motion rendition expected by the animator. The model is based on a simple kinematic approach designed to keep the intrinsic dynamic characteristics of the experimental model. Such an approach also allows a personification of the walking action in an interactive real-time context in most cases. A correction automata of such motion is then proposed.

View PDFchevron_right
Real-Time Realistic Rendering and Subtle Body Movements for Cultural Heritage Virtual Actors
George Papagiannakis

miralab.ch

View PDFchevron_right
Attracted by light: vision-based steering virtual characters among dark and light obstacles
Axel López

Motion, Interaction and Games, 2019

This paper introduces the use of numerical optical flow (OF) in vision-based steering techniques-that control characters locomotion trajectories by using a simulation of their visual perception. In contrast with synthetic OF that was previously used, numerical OF is sensitive to the contrast of objects, and provides, for example, uncertain results in dark areas. Thus, we here propose a locomotion control technique which is robust to such uncertainty: dark areas in the scene are processed as obstacles, that however may be traversed in case of necessity. As demonstrated in various scenarios, this tends to make character avoiding darkest areas, or traversing them more carefully, as it can be observed for real humans. CCS CONCEPTS • Computing methodologies → Modeling and simulation; Model development and analysis; Model verification and validation; Animation.

View PDFchevron_right
Interactive simulation of stylized human locomotion
Marcos AB

2008

Animating natural human motion in dynamic environments is difficult because of complex geometric and physical interactions. Simulation provides an automatic solution to parts of this problem, but it needs control systems to produce lifelike motions. This paper describes the systematic computation of controllers that can reproduce a range of locomotion styles in interactive simulations. Given a reference motion that describes the desired style, a derived control system can reproduce that style in simulation and in new environments. Because it produces high-quality motions that are both geometrically and physically consistent with simulated surroundings, interactive animation systems could begin to use this approach along with more established kinematic methods.

View PDFchevron_right

References (36)

  1. BLACK, M. J., AND ANANDAN, P. 1993. A framework for the robust estimation of optical flow. In Fourth International Conf. on Computer Vision, 231-236.
  2. BUEHLER, C., BOSSE, M., McMILLAN, L., GORTLER, S. J., COHEN, M. F. 2001. Unstructured Lumigraph Rendering. In Proceedings of ACM SIGGRAPH 2001 (Aug 2001), Computer Graphics Proceedings, Annual Conference Series, pp. 425-432.
  3. CARRANZA, J., THEOBALT, C., MAGNOR, M. A., AND SEIDEL, H.-P. 2003. Free-viewpoint video of human actors. ACM Transactions on Graphics 22, 3 (July), 569-577.
  4. CHEN, S. E., AND WILLIAMS, L. 1993. View interpolation for image synthesis. In Proceedings of SIGGRAPH 93, Computer Graphics Proceedings, Annual Conference Series, 279-288.
  5. DEBEVEC, P. E., TAYLOR, C. J., AND MALIK, J. 1996. Modeling and rendering architecture from photographs: A hybrid geometry- and image-based approach. In Proceedings of SIGGRAPH 96, Computer Graphics Proceedings, Annual Conference Series, 11- 20.
  6. DEBEVEC, P., HAWKINS, T., TCHOU, C., DUIKER, H.-P., SAROKIN, W., AND SAGAR, M. 2000. Acquiring the reflectance field of a human face. Proceedings of SIGGRAPH 2000 (July), 145-156.
  7. DORSEY, J., ARVO, J., GREENBERG, D. 1995. Interactive Design of complex time dependent lighting. IEEE Computer Graphics and Applications 15, 2 (Mar. 1995), 26-36
  8. EINARSSON, P., CHABERT, C-F., JONES, A., MA, W-C., LAMOND, B., HAWKINS, T., BOLAS, M., SYLWAN, S., DEBEVEC, P. 2006. Relighting Human Locomotion with Flowed Reflectance Fields. In Eurographics Symposium on Rendering: 17 th Eurographics Workshop on Rendering.
  9. GORTLER, S. J., GRZESZCZUK, R., SZELISKI, R., AND COHEN, M. F. 1996. The lumigraph. In Proceedings of SIGGRAPH 96, Computer Graphics Proceedings, Annual Conference Series, 43- 54.
  10. HAWKINS, T., WENGER, A., TCHOU, C., AND DEBEVEC, A. G. F. G. P. 2004. Animatable facial reflectance fields. In Eurographics Symposium on Rendering: 15th Eurographics Workshop on Rendering.
  11. LAVEAU, S., AND FAUGERAS, O. 1994. 3-D scene representation as a collection of images. In Proceedings of 12th International Conference on Pattern Recognition, vol. 1, 689-691.
  12. LEVOY, M., AND HANRAHAN, P. 1996. Light field rendering. In Proc. of SIGGRAPH 96, Computer Graphics Proceedings, Annual Conference Series, 31-42.
  13. MATUSIK, W., BUEHLER, C., RASKAR, R., GORTLER, S. J., AND MCMILLAN, L. 2000. Image-based visual hulls. In Proc. SIGGRAPH 2000, 369-374.
  14. MATUSIK, W., PFISTER, H., NGAN, A., BEARDSLEY, P., ZIEGLER, R., AND MCMILLAN, L. 2002. Image-based 3d photography using opacity hulls. ACM Transactions on Graphics 21, 3 (July), 427-437.
  15. MATUSIK, W., PFISTER, H., ZIEGLER, R., NGAN, A., AND MCMILLAN, L. 2002a. Acquisition and rendering of transparent and refractive objects. In Rendering Techniques 2002: 13th Eurographics Workshop on Rendering, 267-278.
  16. MCMILLAN, L., AND BISHOP, G. 1995. Plenoptic modeling: An image-based rendering system. In Proceedings of SIGGRAPH 95, Computer Graphics Proceedings, Annual Conference Series, 39- 46.
  17. MEYER, A., NEYRET, F., AND POULIN, P. 2001. Interactive rendering of trees with shading and shadows. In Rendering Techniques 2001: 12th Eurographics Workshop on Rendering, 183-196.
  18. MILLER, G. S. P., RUBIN, S., AND PONCELEON, D. 1998. Lazy decompression of surface light fields for precomputed global illumination. Eurographics Rendering Workshop 1998 (June), 281-292.
  19. MOEZZI, S., KATKERE, A., KURAMURA, D. Y., AND JAIN, R. 1996. Reality modeling and visualization from multiple video sequences. IEEE Computer Graphics & Applications 16, 6 (Nov.), 58-63.
  20. NIMEROFF, J. S., SIMONCELLI, E., AND DORSEY, J. 1994. Efficient re-rendering of naturally illuminated environments. In 5th Eurographics Workshop on Rendering, 359-373.
  21. PORTER, T., AND DUFF, T. 1984. Compositing digital images. In Computer Graphics (Proceedings of SIGGRAPH 84), vol. 18, 253-259.
  22. RAMAMOORTHI, R., AND HANRAHAN, P. 2002. Frequency space environment map rendering. ACM Transactions on Graphics 21, 3 (July), 517-526.
  23. RANDER, P., NARAYANAN, P. J., AND KANADE, T. 1997. Virtualized reality: Constructing time-varying virtual worlds from real events. In Proceedings of IEEE Visualization, 277-283.
  24. SEITZ, S. M., AND DYER, C. R. 1996. View morphing: Synthesizing 3d metamorphoses using image transforms. In Proceedings of SIGGRAPH 96, Computer Graphics Proceedings, Annual Conference Series, 21-30.
  25. SLOAN, P.-P., KAUTZ, J., AND SNYDER, J. 2002. Precomputed radiance transfer for real-time rendering in dynamic, low- frequency lighting environments. ACM Transactions on Graphics 21, 3 (July), 527-536.
  26. SZELISKI, R. 1993. Rapid octree construction from image sequences. CVGIP: Image Understanding 58, 1 (July), 23-32.
  27. THEOBALT, C., AHMED, N., DE AGUIAR, E., ZIEGLER, G., LENSCH, H., MAGNOR, M., AND SEIDEL, H.-P. 2005. Joint motion and reflectance capture for creating relightable 3d videos. Technical Report MPI-I-2005-4-004, Max-Planck-Institut fuer Informatik.
  28. VEDULA, S., BAKER, S., KANADE, T. 2002. Spatio-temporal view interpolation. In Rendering Techniques 2003: 13 th Eurographics Workshop on Rendering (June 2002), pp. 65-76.
  29. VEDULA, S., BAKER, S., KANADE, T. 2005. Image-based spatio- temporal modeling and view interpolation of dynamic events. ACM Transactions on Graphics 24, 2 (Apr. 2005), 240-261.
  30. WENGER, A., GARDNER, A., TCHOU, C., UNGER, J., HAWKINS, T., AND DEBEVEC, P. 2005. Performance relighting and reflectance transformation with time-multiplexed illumination. ACM Transactions on Graphics 24, 3 (Aug.), 756-764.
  31. WILBURN, B., JOSHI, N., VAISH, V., TALVALA, E.-V., ANTUNEZ, E., BARTH, A., ADAMS, A., HOROWITZ, M., AND LEVOY, M. 2005. High performance imaging using large camera arrays. ACM Transactions on Graphics 24, 3 (Aug.), 765-776.
  32. YANG, J. C., EVERETT, M., BUEHLER, C., AND MCMILLAN, L. 2002. A real-time distributed light field camera. In Rendering Techniques 2002: 13th Eurographics Workshop on Rendering, 77- 86.
  33. YU, J., McMILLAN, L., GORTLER, S. 2002. Scan light field rendering. In Pacific Graphics (Beijing, China, Oct. 2002).
  34. ZHANG, L., SNAVELY, N., CURLESS, B., AND SEITZ, S. M. 2004. Spacetime faces: high resolution capture for modeling and animation. ACM Transactions on Graphics 23, 3 (Aug.), 548-558.
  35. ZHANG, Z. 2000. A flexible new technique for camera calibration. PAMI 22, 11, 1330-1334.
  36. ZITNICK, C. L., KANG, S. B., UYTTENDAELE, M., WINDER, S., AND SZELISKI, R. 2004. High-quality video view interpolation using a layered representation. ACM Transactions on Graphics 23, 3 (Aug.), 600-608.

Related papers

Eurographics Symposium on Rendering (2006) Tomas Akenine-Möller and Wolfgang Heidrich (Editors) Relighting Human Locomotion with Flowed Reflectance
Sebastian Sylwan

2014

We present an image-based approach for capturing the appearance of a walking or running person so they can be rendered realistically under variable viewpoint and illumination. In our approach, a person walks on a treadmill at a regular rate as a turntable slowly rotates the person’s direction. As this happens, the person is filmed with a vertical array of high-speed cameras under a time-multiplexed lighting basis, acquiring a seven-dimensional dataset of the person under variable time, illumination, and viewing direction in approximately forty seconds. We process this data into a flowed reflectance field using an optical flow algorithm to correspond pixels in neighboring camera views and time samples to each other, and we use image compression to reduce the size of this data. We then use image-based relighting and a hardware-accelerated combination of view morphing and light field rendering to render the subject under user-specified viewpoint and lighting conditions. To composite th...

View PDFchevron_right
Relighting Human Locomotion with Flowed Reflectance Fields
Sebastian Sylwan

2006

We present an image-based approach for capturing the appearance of a walking or running person so they can be rendered realistically under variable viewpoint and illumination. In our approach, a person walks on a treadmill at a regular rate as a turntable slowly rotates the person's direction. As this happens, the person is filmed with a vertical array of high-speed cameras under a time-multiplexed lighting basis, acquiring a seven-dimensional dataset of the person under variable time, illumination, and viewing direction in approximately forty seconds. We process this data into a flowed reflectance field using an optical flow algorithm to correspond pixels in neighboring camera views and time samples to each other, and we use image compression to reduce the size of this data. We then use image-based relighting and a hardware-accelerated combination of view morphing and light field rendering to render the subject under user-specified viewpoint and lighting conditions. To composite the person into a scene, we use an alpha channel derived from back lighting and a retroreflective treadmill surface and a visual hull process to render the shadows the person would cast onto the ground. We demonstrate realistic composites of several subjects into real and virtual environments using our technique.

View PDFchevron_right
Seeing People in Different Light-Joint Shape, Motion, and Reflectance Capture
naveed ahmed

IEEE Transactions on Visualization and Computer Graphics, 2007

By means of passive optical motion capture, real people can be authentically animated and photo-realistically textured. To import real-world characters into virtual environments, however, surface reflectance properties must also be known. We describe a video-based modeling approach that captures human shape and motion as well as reflectance characteristics from a handful of synchronized video recordings. The presented method is able to recover spatially varying surface reflectance properties of clothes from multiview video footage. The resulting model description enables us to realistically reproduce the appearance of animated virtual actors under different lighting conditions, as well as to interchange surface attributes among different people, e.g., for virtual dressing. Our contribution can be used to create 3D renditions of real-world people under arbitrary novel lighting conditions on standard graphics hardware.

View PDFchevron_right
Real-Time Realistic Rendering and Subtle Body Movements for Virtual Actors
George Papagiannakis

george.papagiannakis.org

View PDFchevron_right
Vision-based animation of digital humans
Ioannis Kakadiaris

1998

This paper presents a system for animating customized virtual humans using motion parameters estimated from multi-view image sequences. The advantage of our method is that the subject does not have to wear markers or other devices.

View PDFchevron_right
580 California St., Suite 400
San Francisco, CA, 94104
© 2025 Academia. All rights reserved