Academia.eduAcademia.edu

Figure 32 – uploaded by P. Eisert

See full PDFdownloadDownload figure
Figure 4.2: a) Three frames of the sequence Peter, b) corresponding synthetic views. 4.2. Analysis and Synthesis of Head-and-Shoulder Sequences

Figure 4 2: a) Three frames of the sequence Peter, b) corresponding synthetic views. 4.2. Analysis and Synthesis of Head-and-Shoulder Sequences

Related Figures (77)

Figure 2.1: Structure of a model-based codec.
Figure 2.1: Structure of a model-based codec.
This depth map is obtained by rendering the 3-D object model into a z-buffer. Setting x. = [0 0 OJ and f, = f, = 1 reduces (3.9) to the well-known motior constraint in [LH84, NS85, WKPS87, LRF93]. This corresponds to a linearization o the motion model around the camera origin instead of linearization at the object center However, the linearization error for the reduced case is slightly larger. This is illustratec in Fig. 3.1 where the average absolute displacement error for different rotation angles is plotted for the object shown in Fig. 3.4 b). ] Especially for larger rotation angles significan differences in accuracy are noticeable. At the same time, neither the number of equation: nor the number of unknowns and thus the complexity of solving the system of equations as described in the next section is affected by the improved linearization method.
This depth map is obtained by rendering the 3-D object model into a z-buffer. Setting x. = [0 0 OJ and f, = f, = 1 reduces (3.9) to the well-known motior constraint in [LH84, NS85, WKPS87, LRF93]. This corresponds to a linearization o the motion model around the camera origin instead of linearization at the object center However, the linearization error for the reduced case is slightly larger. This is illustratec in Fig. 3.1 where the average absolute displacement error for different rotation angles is plotted for the object shown in Fig. 3.4 b). ] Especially for larger rotation angles significan differences in accuracy are noticeable. At the same time, neither the number of equation: nor the number of unknowns and thus the complexity of solving the system of equations as described in the next section is affected by the improved linearization method.
Figure 3.1: Average absolute displacement error in pixels caused by the linearization of the motion model; a) error of (3.9) for rotation around the x- and y-axes (measured in degrees), b) corresponding errors for the simplified motion model of [LH84].
Figure 3.1: Average absolute displacement error in pixels caused by the linearization of the motion model; a) error of (3.9) for rotation around the x- and y-axes (measured in degrees), b) corresponding errors for the simplified motion model of [LH84].
The optical flow constraint (3.11) can be set up for any pixel in the image. However since the two unknown components d, and d, of the displacement vector vary from pixe to pixel, the flow constraint is underdetermined and cannot be solved directly without additional information. This information is taken directly from the explicit 2-D motior model (3.9) that describes the displacement vector d for all image points containing the object as a function of the 6 unknown rigid body parameters. Inserting (3.9) into the optical flow constraint (3.11) leads to a set of linear equations for the unknowns AR, AR,, AR,, At,, Aty, and At,
The optical flow constraint (3.11) can be set up for any pixel in the image. However since the two unknown components d, and d, of the displacement vector vary from pixe to pixel, the flow constraint is underdetermined and cannot be solved directly without additional information. This information is taken directly from the explicit 2-D motior model (3.9) that describes the displacement vector d for all image points containing the object as a function of the 6 unknown rigid body parameters. Inserting (3.9) into the optical flow constraint (3.11) leads to a set of linear equations for the unknowns AR, AR,, AR,, At,, Aty, and At,
The rigid body motion estimator of the previous section estimates the motion parame- ters of an object between two successive frames of a video sequence. To overcome the restrictions from the linearization of small motion between these frames, the algorithm is embedded in an analysis-synthesis loop [Koc93, LRF93] as depicted in Fig. 3.2. Instead of Figure 3.2: Analysis-synthesis loop of the motion estimation framework.
The rigid body motion estimator of the previous section estimates the motion parame- ters of an object between two successive frames of a video sequence. To overcome the restrictions from the linearization of small motion between these frames, the algorithm is embedded in an analysis-synthesis loop [Koc93, LRF93] as depicted in Fig. 3.2. Instead of Figure 3.2: Analysis-synthesis loop of the motion estimation framework.
Figure 3.3: Hierarchical estimation scheme. is applied to a synthetically rendered image and the new camera frame. At the begin- ning of the estimation, the motion parameter set corresponding to the previous frame [, is used to render an estimate I for the current camera frame I. The relative motion between these frames is determined as described above using the shape information from the 3-D model. Then, the 3-D model is moved according to the new motion parameter set and a new motion compensated frame J is rendered that is now much closer to the camera frame Jy than the previous estimate. This procedure is performed iteratively, each time reducing the remaining estimation errors. The whole scheme can be regarded as an opti- mization procedure t hat finds the best match between the synthetic frame and the camera frame. Due to the use of a synthetic frame that directly corresponds to the 3-D model providing the algorit hm with shape information, error accumulation over time is avoided, since remaining errors are removed in the following iteration steps. Thus, external forces from discrete image eatures as, e.g., used in [DM96] to avoid error accumulation can be
Figure 3.3: Hierarchical estimation scheme. is applied to a synthetically rendered image and the new camera frame. At the begin- ning of the estimation, the motion parameter set corresponding to the previous frame [, is used to render an estimate I for the current camera frame I. The relative motion between these frames is determined as described above using the shape information from the 3-D model. Then, the 3-D model is moved according to the new motion parameter set and a new motion compensated frame J is rendered that is now much closer to the camera frame Jy than the previous estimate. This procedure is performed iteratively, each time reducing the remaining estimation errors. The whole scheme can be regarded as an opti- mization procedure t hat finds the best match between the synthetic frame and the camera frame. Due to the use of a synthetic frame that directly corresponds to the 3-D model providing the algorit hm with shape information, error accumulation over time is avoided, since remaining errors are removed in the following iteration steps. Thus, external forces from discrete image eatures as, e.g., used in [DM96] to avoid error accumulation can be
Table 3.1: Experimental setup for the synthetic test sequence. RGB color images as shown in Fig. 3.4 c). Then, the motion parameters are estimated for each frame. Four different levels of resolution are used by the hierarchical algorithm, starting with images of size 44 x 36 and ending with CIF resolution. Since the correct motion parameters are known for the synthetic sequence, the accuracy of the motion estimation is determined by comparing the estimates with the correct values. Din 2K aghacra tha ahanlatoan nrrnar far panh matinn naramatnar arnarag~an nrvaor all INN
Table 3.1: Experimental setup for the synthetic test sequence. RGB color images as shown in Fig. 3.4 c). Then, the motion parameters are estimated for each frame. Four different levels of resolution are used by the hierarchical algorithm, starting with images of size 44 x 36 and ending with CIF resolution. Since the correct motion parameters are known for the synthetic sequence, the accuracy of the motion estimation is determined by comparing the estimates with the correct values. Din 2K aghacra tha ahanlatoan nrrnar far panh matinn naramatnar arnarag~an nrvaor all INN
Figure 3.5: Absolute error of the estimated motion parameter averaged over 100 frames of a synthetic sequence with additive Gaussian noise with standard deviation of 0, 2, 5, 10, 20, and 50, respectively. 3. Model-Based Estimation of 3-D Motion and Deformation
Figure 3.5: Absolute error of the estimated motion parameter averaged over 100 frames of a synthetic sequence with additive Gaussian noise with standard deviation of 0, 2, 5, 10, 20, and 50, respectively. 3. Model-Based Estimation of 3-D Motion and Deformation
Figure 3.6: Absolute error of the estimated motion parameter averaged over 100 frames using a 3-D model with distorted vertex positions. 3.1. Rigid Body Motion Estimation
Figure 3.6: Absolute error of the estimated motion parameter averaged over 100 frames using a 3-D model with distorted vertex positions. 3.1. Rigid Body Motion Estimation
Figure 3.7: Absolute error of the estimated motion parameter averaged over 100 frames using a model with distorted vertex positions and texture extracted from an initial frame.
Figure 3.7: Absolute error of the estimated motion parameter averaged over 100 frames using a model with distorted vertex positions and texture extracted from an initial frame.
parameter estimation of ARz, ARy, ARz, Atz, At,, Atz, and Ak; performed using those two frames Figure 3.8: Estimation of radial lens distortion and rigid body motion. with the radius r computed as follows
parameter estimation of ARz, ARy, ARz, Atz, At,, Atz, and Ak; performed using those two frames Figure 3.8: Estimation of radial lens distortion and rigid body motion. with the radius r computed as follows
This displacement field is quite similar to the one for rigid body motion (3.9) but contains the additional unknowns Af, Ag, and Ax«;. As a result, by combining (3.22) with the into (3.18) leads, after first order Taylor series expansion, to the following displacement Inserting (3.20) together with the replacement for the 3-D coordinates
This displacement field is quite similar to the one for rigid body motion (3.9) but contains the additional unknowns Af, Ag, and Ax«;. As a result, by combining (3.22) with the into (3.18) leads, after first order Taylor series expansion, to the following displacement Inserting (3.20) together with the replacement for the 3-D coordinates
Many algorithms in computer vision assume that the internal parameters of the camera are known. Therefore, camera calibration is performed in an initial step to determine focal length, aspect ratio, and radial lens distortion of the camera. A calibration test object as shown in Fig. 3.9 is captured with the camera. Since the geometry of the test Figure 3.9: Two different calibration test objects used in the experiments of this thesis.
Many algorithms in computer vision assume that the internal parameters of the camera are known. Therefore, camera calibration is performed in an initial step to determine focal length, aspect ratio, and radial lens distortion of the camera. A calibration test object as shown in Fig. 3.9 is captured with the camera. Since the geometry of the test Figure 3.9: Two different calibration test objects used in the experiments of this thesis.
Table 3.3: Experimental setup for the synthetic test sequence. ~ — ~*~ To quantify the accuracy of the algorithm, a synthetic sequence is rendered using the calibration test object shown on the left hand side of Fig. 3.9. The experimental setup is specified in Table 3.3. Different sets of internal and external camera parameters are chosen to synthesize the frames. To simulate manufacturing errors of the calibration body, the 2-D positions of the white points on the test object planes are distorted by adding zero mean normally distributed noise with standard deviation o to the point locations before rendering the 3-D model. Given the synthetic frames corresponding to different before rendering the 3-D model. Given the synthetic frames corresponding to different
Table 3.3: Experimental setup for the synthetic test sequence. ~ — ~*~ To quantify the accuracy of the algorithm, a synthetic sequence is rendered using the calibration test object shown on the left hand side of Fig. 3.9. The experimental setup is specified in Table 3.3. Different sets of internal and external camera parameters are chosen to synthesize the frames. To simulate manufacturing errors of the calibration body, the 2-D positions of the white points on the test object planes are distorted by adding zero mean normally distributed noise with standard deviation o to the point locations before rendering the 3-D model. Given the synthetic frames corresponding to different before rendering the 3-D model. Given the synthetic frames corresponding to different
Figure 3.10: Average absolute error of the estimated internal and external camera parameters for test objects of varying accuracy. Aspect ratio a and height angle Ypeignt are specified relative to their mean @ = 4 and Pneight = 0.5, respectively. 3.2. Combined Estimation of Motion and Camera Parameters
Figure 3.10: Average absolute error of the estimated internal and external camera parameters for test objects of varying accuracy. Aspect ratio a and height angle Ypeignt are specified relative to their mean @ = 4 and Pneight = 0.5, respectively. 3.2. Combined Estimation of Motion and Camera Parameters
Figure 3.11: Frames of the test object from different viewing positions. 3. Model-Based Estimation of 3-D Motion and Deformation
Figure 3.11: Frames of the test object from different viewing positions. 3. Model-Based Estimation of 3-D Motion and Deformation
Figure 3.12: Standard deviation of the internal camera parameters for different numbers of frames used simultaneously in the estimation.
Figure 3.12: Standard deviation of the internal camera parameters for different numbers of frames used simultaneously in the estimation.
Figure 3.13: Illumination estimation and compensation scheme.
Figure 3.13: Illumination estimation and compensation scheme.
and sparse and thus efficient algorithms for the inversion are available |GL83}. qairections that ao not alrectly colncide with the elements OF tne Map. For the estimation of reflectance maps, (3.33) is solved for the unknown map entries R°(nz,Ny). For nearest neighbor interpolation, the pixels can easily be classified according to their normals and each entry of the reflectance map is estimated independently of the others. Higher order interpolation leads to a coupling of the unknowns and a linear syste of equations has to be solved. Fortunately, the resulting matrix to be inverted is bandec and sparse and thus efficient algorithms for the inversion are available [GL83].
and sparse and thus efficient algorithms for the inversion are available |GL83}. qairections that ao not alrectly colncide with the elements OF tne Map. For the estimation of reflectance maps, (3.33) is solved for the unknown map entries R°(nz,Ny). For nearest neighbor interpolation, the pixels can easily be classified according to their normals and each entry of the reflectance map is estimated independently of the others. Higher order interpolation leads to a coupling of the unknowns and a linear syste of equations has to be solved. Fortunately, the resulting matrix to be inverted is bandec and sparse and thus efficient algorithms for the inversion are available [GL83].
The performance of the different approaches are compared in Table 3.4. The param- eters of all reflection and lighting models are estimated for all frames and the synthetic frames are shaded appropriately. The resulting model frames are compared with the orig- inal frames in terms of PSNR measured inside the object area. The black background is excluded from the measurement. Table 3.4 shows the resulting PSNR for the differ- ent approaches averaged over all frames. Even for the Lambert reflection model, the ent approaches averaged over all frames. Even for the Lambert reflection model, the
The performance of the different approaches are compared in Table 3.4. The param- eters of all reflection and lighting models are estimated for all frames and the synthetic frames are shaded appropriately. The resulting model frames are compared with the orig- inal frames in terms of PSNR measured inside the object area. The black background is excluded from the measurement. Table 3.4 shows the resulting PSNR for the differ- ent approaches averaged over all frames. Even for the Lambert reflection model, the ent approaches averaged over all frames. Even for the Lambert reflection model, the
3. Model-Based Estimation of 3-D Motion and Deformation Figure 3.15: Upper row: video frames of an artificial head object illuminated in different ways, lower row: corresponding model frames after lighting adaptation with 2nd order reflection model.
3. Model-Based Estimation of 3-D Motion and Deformation Figure 3.15: Upper row: video frames of an artificial head object illuminated in different ways, lower row: corresponding model frames after lighting adaptation with 2nd order reflection model.
tion is then estimated as described in Section 3.3.1 and again compared with the original vectors. As shown in the first row of Table 3.5, two iterations in the analysis-synthesis loop are sufficient to compensate also for large changes in the illuminant direction. After the third iteration, no further changes are observable. If the maximum angle between the correct illuminant direction and the initial value is less than 45° or 30°, respectively, the convergence is even faster as illustrated in the lower two rows of Table 3.5. convergence is even faster as illustrated in the lower two rows of Table 3.5.
tion is then estimated as described in Section 3.3.1 and again compared with the original vectors. As shown in the first row of Table 3.5, two iterations in the analysis-synthesis loop are sufficient to compensate also for large changes in the illuminant direction. After the third iteration, no further changes are observable. If the maximum angle between the correct illuminant direction and the initial value is less than 45° or 30°, respectively, the convergence is even faster as illustrated in the lower two rows of Table 3.5. convergence is even faster as illustrated in the lower two rows of Table 3.5.
In the next experiment, a real world video sequence consisting of 50 frames is recorded. The illumination setup is held constant during the acquisition but the head object rotates around its y-axis leading to varying shading effects at the object surface. Fig. 3.4 a) depicts the first frame of this sequence that is used to estimate the illumination differences between the synthetic and the real scene. The resulting illumination parameters are ther used to compensate all following frames of the video sequence. PSNR between the model and the camera frame is computed for the facial area of the images. Table 3.6 shows the average reconstruction quality for the different approaches. Even for this experiment. Table 3.6: Average reconstruction quality of the model frames shaded with different reflection models. the average reconstruction quality for the different approaches. Even for this experiment,
In the next experiment, a real world video sequence consisting of 50 frames is recorded. The illumination setup is held constant during the acquisition but the head object rotates around its y-axis leading to varying shading effects at the object surface. Fig. 3.4 a) depicts the first frame of this sequence that is used to estimate the illumination differences between the synthetic and the real scene. The resulting illumination parameters are ther used to compensate all following frames of the video sequence. PSNR between the model and the camera frame is computed for the facial area of the images. Table 3.6 shows the average reconstruction quality for the different approaches. Even for this experiment. Table 3.6: Average reconstruction quality of the model frames shaded with different reflection models. the average reconstruction quality for the different approaches. Even for this experiment,
Figure 3.16: Average reconstruction quality for different sizes of the reflectance map. 3. Model-Based Estimation of 3-D Motion and Deformation
Figure 3.16: Average reconstruction quality for different sizes of the reflectance map. 3. Model-Based Estimation of 3-D Motion and Deformation
Figure 3.17: Absolute motion estimation error averaged over 50 frames. The two cases for simple motion estimation assuming ambient illumination and combined motion and illumination estimation (Lambert reflection) are shown. performed but now both motion and illumination parameters are estimated alternately in the hierarchical analysis-synthesis loop shown in Fig. 3.2. From 50 calibrated video frames with constant illumination, the motion parameters are estimated and compared with the original ones. Fig. 3.17 shows the average absolute error of the six rigid body
Figure 3.17: Absolute motion estimation error averaged over 50 frames. The two cases for simple motion estimation assuming ambient illumination and combined motion and illumination estimation (Lambert reflection) are shown. performed but now both motion and illumination parameters are estimated alternately in the hierarchical analysis-synthesis loop shown in Fig. 3.2. From 50 calibrated video frames with constant illumination, the motion parameters are estimated and compared with the original ones. Fig. 3.17 shows the average absolute error of the six rigid body
Figure 3.18: a) topology of the triangular B-spline patches, b) individually adapted triangle mesh used for rendering and motion estimation, c) control points specifying the shape of the textured surface.
Figure 3.18: a) topology of the triangular B-spline patches, b) individually adapted triangle mesh used for rendering and motion estimation, c) control points specifying the shape of the textured surface.
Figure 3.19: Modeling of eye movements, a) with blending, b) without blending.
Figure 3.19: Modeling of eye movements, a) with blending, b) without blending.
hair if viewed from a certain viewing direction are colored appropriately; the remainin; texels are set to be transparent. The color information for this additional plane can b: directly taken from a video frame since the internal camera parameters specifying th perspective projection are known from the virtual scene. This way, the silhouette of th hair is accurately recovered if the viewing direction is not changed too much. Other wise, multiple planes can be used to obtain a natural appearance for arbitrary viewin, directions. Fig. 3.21 shows some results obtained with this technique.
hair if viewed from a certain viewing direction are colored appropriately; the remainin; texels are set to be transparent. The color information for this additional plane can b: directly taken from a video frame since the internal camera parameters specifying th perspective projection are known from the virtual scene. This way, the silhouette of th hair is accurately recovered if the viewing direction is not changed too much. Other wise, multiple planes can be used to obtain a natural appearance for arbitrary viewin, directions. Fig. 3.21 shows some results obtained with this technique.
Figure 3.21: Different facial expressions generated by deforming the surface of a 3-D head model.
Figure 3.21: Different facial expressions generated by deforming the surface of a 3-D head model.
Figure 3.22: Transformations from FAPs to image points.
Figure 3.22: Transformations from FAPs to image points.
Figure 3.23: a) undistorted model frame, b) synthetic frame with additive noise, c) synthetic frame with distorted control point positions.
Figure 3.23: a) undistorted model frame, b) synthetic frame with additive noise, c) synthetic frame with distorted control point positions.
Figure 3.24: Absolute error of the estimated motion parameters averaged over 100 frames of a synthetic sequence with additive Gaussian noise (standard deviation 0, 2, 5, 10, 20, and 50, respectively).
Figure 3.24: Absolute error of the estimated motion parameters averaged over 100 frames of a synthetic sequence with additive Gaussian noise (standard deviation 0, 2, 5, 10, 20, and 50, respectively).
3. Model-Based Estimation of 3-D Motion and Deformation Figure 3.25: Absolute error of the estimated motion parameters averaged over 100 frames. The frames of the sequence are rendered from a 3-D model with distorted control point positions.
3. Model-Based Estimation of 3-D Motion and Deformation Figure 3.25: Absolute error of the estimated motion parameters averaged over 100 frames. The frames of the sequence are rendered from a 3-D model with distorted control point positions.
Table 4.1: Video sequences used for the experiments. 4. Model-Based Coding of Head-and-Shoulder Sequences
Table 4.1: Video sequences used for the experiments. 4. Model-Based Coding of Head-and-Shoulder Sequences
Figure 4.3: PSNR for the sequence Peter for the two cases that, first, only global motion of the head and, second, both global and local motion is estimated. 4. Model-Based Coding of Head-and-Shoulder Sequences
Figure 4.3: PSNR for the sequence Peter for the two cases that, first, only global motion of the head and, second, both global and local motion is estimated. 4. Model-Based Coding of Head-and-Shoulder Sequences
Figure 4.4: a) Three frames of the sequence Eckehard, b) corresponding synthetic views.
Figure 4.4: a) Three frames of the sequence Eckehard, b) corresponding synthetic views.
Figure 4.5: a) Three frames of sequence Akiyo, b) corresponding synthetic views.
Figure 4.5: a) Three frames of sequence Akiyo, b) corresponding synthetic views.
Figure 4.6: Temporal evolution of Camp and cq; for sequence Illumination. 4. Model-Based Coding of Head-and-Shoulder Sequences
Figure 4.6: Temporal evolution of Camp and cq; for sequence Illumination. 4. Model-Based Coding of Head-and-Shoulder Sequences
Table 4.2: Reconstruction quality of the sequence Illumination for different lighting models. circumvent this problem, the estimation of illumination parameters is performed in two steps. First, a set of parameters is determined describing differences between the texture map of the model acquired under controlled illumination and the first camera frame. Given the estimated values, the shading effects of the camera frame are removed by applying the inverse of the relation (3.27) before mapping the camera frame onto the model. Since the corrected texture map corresponds to a homogeneously illuminated frame, the estimation of photometric properties remains linear and is performed following the description in Section 3.3.
Table 4.2: Reconstruction quality of the sequence Illumination for different lighting models. circumvent this problem, the estimation of illumination parameters is performed in two steps. First, a set of parameters is determined describing differences between the texture map of the model acquired under controlled illumination and the first camera frame. Given the estimated values, the shading effects of the camera frame are removed by applying the inverse of the relation (3.27) before mapping the camera frame onto the model. Since the corrected texture map corresponds to a homogeneously illuminated frame, the estimation of photometric properties remains linear and is performed following the description in Section 3.3.
4. Model-Based Coding of Head-and-Shoulder Sequences Figure 4.7: a) Three frames of the sequence Illumination recorded under varying illumination conditions, b) corresponding synthetic views for the 8 parameter Lambertian illumination model.
4. Model-Based Coding of Head-and-Shoulder Sequences Figure 4.7: a) Three frames of the sequence Illumination recorded under varying illumination conditions, b) corresponding synthetic views for the 8 parameter Lambertian illumination model.
Figure 4.8: Extracted model of the clapper board.
Figure 4.8: Extracted model of the clapper board.
Figure 4.9: a) Three frames of sequence Clapper Board, b) corresponding synthetic views. the entire object. For that purpose, the clapper board is segmented from the backgrounc and the obtained region is automatically triangulated as depicted in Fig. 4.8. The joint o the clapper board that allows opening it is specified manually. Five animation parameter: determine pose and shape of the object: translation in three directions, rotation in the image plane and opening of the clapper board. These five parameters are estimated ir addition to the FAPs of the head model using two independent motion estimators. Fig. 4.‘ shows three frames of the sequence together with the corresponding model frames createc by rendering the two objects with the estimated parameters. The leftmost image in this figure represents the decoded frame with the worst PSNR (27.2 dB) in the entire sequence
Figure 4.9: a) Three frames of sequence Clapper Board, b) corresponding synthetic views. the entire object. For that purpose, the clapper board is segmented from the backgrounc and the obtained region is automatically triangulated as depicted in Fig. 4.8. The joint o the clapper board that allows opening it is specified manually. Five animation parameter: determine pose and shape of the object: translation in three directions, rotation in the image plane and opening of the clapper board. These five parameters are estimated ir addition to the FAPs of the head model using two independent motion estimators. Fig. 4.‘ shows three frames of the sequence together with the corresponding model frames createc by rendering the two objects with the estimated parameters. The leftmost image in this figure represents the decoded frame with the worst PSNR (27.2 dB) in the entire sequence
Figure 4.10: PSNR for sequence Clapper Board for the two cases that, first, only facial motion and, second, animation parameters for both the face and the additional clapper board are estimated.
Figure 4.10: PSNR for sequence Clapper Board for the two cases that, first, only facial motion and, second, animation parameters for both the face and the additional clapper board are estimated.
Figure 4.11: Reconstruction quality in PSNR over bit-rate needed for encoding the animation parameters; a) sequence Peter, b) sequence Eckehard. AVLLLS All ALLvbilitsovuiwe CULL VeiViv MMV tELS VLA UUU MA YVLEL ULL ULLAL. In a first experiment, the 18 FAPs estimated as described in Section 4.2 are encoded at 25 fps for the two sequences Peter and Eckehard. The bitstream is decoded and the heac model is animated according to the quantized values synthesizing the sequence again PSNR is measured for this synthetic sequence for different bit-rates obtained by varying the quantizer step size. Fig. 4.11 shows the rate-distortion plots obtained for the model. based codec. Due to model failures in the 3-D head model, the curves saturate at highe1 bit-rates and converge to the quality obtained for unquantized FAPs. The bit-rate needec to come close to that point is around 1 kbps. When using a frame skip of 2 and encoding
Figure 4.11: Reconstruction quality in PSNR over bit-rate needed for encoding the animation parameters; a) sequence Peter, b) sequence Eckehard. AVLLLS All ALLvbilitsovuiwe CULL VeiViv MMV tELS VLA UUU MA YVLEL ULL ULLAL. In a first experiment, the 18 FAPs estimated as described in Section 4.2 are encoded at 25 fps for the two sequences Peter and Eckehard. The bitstream is decoded and the heac model is animated according to the quantized values synthesizing the sequence again PSNR is measured for this synthetic sequence for different bit-rates obtained by varying the quantizer step size. Fig. 4.11 shows the rate-distortion plots obtained for the model. based codec. Due to model failures in the 3-D head model, the curves saturate at highe1 bit-rates and converge to the quality obtained for unquantized FAPs. The bit-rate needec to come close to that point is around 1 kbps. When using a frame skip of 2 and encoding
Table 4.3: Bit-rate needed to encode the FAPs at a reconstruction quality less than 0.1 dB inferior to the unquantized case. The results are given for a frame skip of 0 and 2, respectively.
Table 4.3: Bit-rate needed to encode the FAPs at a reconstruction quality less than 0.1 dB inferior to the unquantized case. The results are given for a frame skip of 0 and 2, respectively.
Figure 4.12: Reconstruction quality in PSNR over bit-rate needed for encoding the FAPs with a frame skip of 2; a) sequence Peter, b) sequence Eckehard. 4. Model-Based Coding of Head-and-Shoulder Sequences
Figure 4.12: Reconstruction quality in PSNR over bit-rate needed for encoding the FAPs with a frame skip of 2; a) sequence Peter, b) sequence Eckehard. 4. Model-Based Coding of Head-and-Shoulder Sequences
Figure 4.13: Reconstruction quality in PSNR over bit-rate needed for encoding the illumina- tion parameters. A bit-rate of zero corresponds to no illumination compensation; a) sequence Illumination, b) sequence Peter with a fairly constant illumination situation.
Figure 4.13: Reconstruction quality in PSNR over bit-rate needed for encoding the illumina- tion parameters. A bit-rate of zero corresponds to no illumination compensation; a) sequence Illumination, b) sequence Peter with a fairly constant illumination situation.
Figure 4.14: upper row: original video sequence. From this sequence, FAPs are estimated and used to synthesize new sequences with different 3-D models. 4. Model-Based Coding of Head-and-Shoulder Sequences
Figure 4.14: upper row: original video sequence. From this sequence, FAPs are estimated and used to synthesize new sequences with different 3-D models. 4. Model-Based Coding of Head-and-Shoulder Sequences
Figure 5.1: Structure of the model-aided coder. Traditional block-based MCP from the previous decoded frame is extended by prediction from the current model frame. 5.2. Rate-Constrained Coder Control
Figure 5.1: Structure of the model-aided coder. Traditional block-based MCP from the previous decoded frame is extended by prediction from the current model frame. 5.2. Rate-Constrained Coder Control
Fig. 5.2 shows the rate-distortion curves obtained for the sequence Peter. Significant gains Figure 5.2: Rate-distortion plot for the sequence Peter. .3.1 Model-Aided Coding of Head-and-Shoulder Sequences
Fig. 5.2 shows the rate-distortion curves obtained for the sequence Peter. Significant gains Figure 5.2: Rate-distortion plot for the sequence Peter. .3.1 Model-Aided Coding of Head-and-Shoulder Sequences
The image on the left hand side of Fig. 5.3 is decoded and reconstructed from the TMN-10 decoder, while the frame on the right hand side originates from the model-aided coder. Both frames require about the same number of bits and are taken from sequences that are approximately encoded at the same average bit-rate using a DCT quantizer parameter of 31 for the TMN-10 and a value of 25 for the model-aided coder.
The image on the left hand side of Fig. 5.3 is decoded and reconstructed from the TMN-10 decoder, while the frame on the right hand side originates from the model-aided coder. Both frames require about the same number of bits and are taken from sequences that are approximately encoded at the same average bit-rate using a DCT quantizer parameter of 31 for the TMN-10 and a value of 25 for the model-aided coder.
Figure 5.4: Rate-distortion plot for the sequence Eckehard.
Figure 5.4: Rate-distortion plot for the sequence Eckehard.
For the first two experiments, an explicit head model created from a 3-D laser scan was available. This is not the case for the next video sequence. Fig. 5.6 shows results for the sequence Akiyo. A generic head model is roughly adapted to the first frame of the sequence (see Section 4.2) and model frames as shown in Fig. 4.5 are rendered from the approximate object model. Even if only an inaccurate 3-D model is used for the generation of the model frames, bit-rate savings of the model-aided codec are still about
For the first two experiments, an explicit head model created from a 3-D laser scan was available. This is not the case for the next video sequence. Fig. 5.6 shows results for the sequence Akiyo. A generic head model is roughly adapted to the first frame of the sequence (see Section 4.2) and model frames as shown in Fig. 4.5 are rendered from the approximate object model. Even if only an inaccurate 3-D model is used for the generation of the model frames, bit-rate savings of the model-aided codec are still about
Table 5.1: Percentage of macroblocks that are selected from the model frame for different quantizer values and video sequences. as many bits. upper right image shows frame 150 encoded with the model- aided coder, while the upper left image corresponds to the TMN-10 coder at the same bit-rate. At the lower right of Fig. 5.7, a frame from the TMN-10 coder is shown that has the same PSNR as the upper model-aided frame. Even though the PSNR is the same, the subjective quality of the reconstructed frame from the model-aided coder is superior since facial features are reproduced more accurately and with fewer artifacts. The dit fference is even more striking when viewing motion sequences. Finally, the lower left image is encoded with TMN-10 to yield the same subjective quality as the model-aided coder; 1] as many bits. MN-10 requires about twice
Table 5.1: Percentage of macroblocks that are selected from the model frame for different quantizer values and video sequences. as many bits. upper right image shows frame 150 encoded with the model- aided coder, while the upper left image corresponds to the TMN-10 coder at the same bit-rate. At the lower right of Fig. 5.7, a frame from the TMN-10 coder is shown that has the same PSNR as the upper model-aided frame. Even though the PSNR is the same, the subjective quality of the reconstructed frame from the model-aided coder is superior since facial features are reproduced more accurately and with fewer artifacts. The dit fference is even more striking when viewing motion sequences. Finally, the lower left image is encoded with TMN-10 to yield the same subjective quality as the model-aided coder; 1] as many bits. MN-10 requires about twice
35 % at low bit-rates. The quality of the reconstructed frames is shown in Fig. 5.7. The Figure 5.6: Rate-distortion plot for the sequence Akiyo.
35 % at low bit-rates. The quality of the reconstructed frames is shown in Fig. 5.7. The Figure 5.6: Rate-distortion plot for the sequence Akiyo.
Table 5.2: Relative frequency of the 4 modes SKIP, INT] model-aided coder. Two different quantizer parameters (31 and 15) Peter (PE), Eckehard (EK), Akiyo (AK), and Illumination (IL). 1 ER, INT) ER-4V, and INTRA for the are chosen for the sequences Che lower three rows of this table depict the coding mode frequency for the case where the model frame is chosen for pre- diction.
Table 5.2: Relative frequency of the 4 modes SKIP, INT] model-aided coder. Two different quantizer parameters (31 and 15) Peter (PE), Eckehard (EK), Akiyo (AK), and Illumination (IL). 1 ER, INT) ER-4V, and INTRA for the are chosen for the sequences Che lower three rows of this table depict the coding mode frequency for the case where the model frame is chosen for pre- diction.
Figure 5.8: Model and camera frame 21 of the sequence Akiyo with enlargement of the image region around the mouth. The impact of the rate-constrained coder control on the reference frame selection is exemplarily shown for one frame of the sequence Akiyo. Fig. 5.8 depicts the model- based prediction of frame 21 generated by rendering the 3-D head model. As can be seen in the enlargement of the region around the mouth, the interior of the mouth is not correctly modeled since this information was not available during model acquisition. The correctly modeled since this information was not available during model acquisition. The
Figure 5.8: Model and camera frame 21 of the sequence Akiyo with enlargement of the image region around the mouth. The impact of the rate-constrained coder control on the reference frame selection is exemplarily shown for one frame of the sequence Akiyo. Fig. 5.8 depicts the model- based prediction of frame 21 generated by rendering the 3-D head model. As can be seen in the enlargement of the region around the mouth, the interior of the mouth is not correctly modeled since this information was not available during model acquisition. The correctly modeled since this information was not available during model acquisition. The
Figure 5.9: Frame 21 of the sequence Akiyo. Macroblocks shown have been selected from the model frame. The grey parts of the image are predicted from the previous frame, or they are encoded in INTRA mode. in Fig. 5.9. Fig. 5.9 shows all macroblocks that are predicted from the model frame, while the macroblocks predicted from the previous frame or encoded in INTRA mode are grey. The mouth is not predicted from the model frame thus avoiding the prediction error coding from the generic teeth in Fig. 5.8. The same applies to macroblocks containing the object silhouette, since the background of the scene is not correctly modeled.
Figure 5.9: Frame 21 of the sequence Akiyo. Macroblocks shown have been selected from the model frame. The grey parts of the image are predicted from the previous frame, or they are encoded in INTRA mode. in Fig. 5.9. Fig. 5.9 shows all macroblocks that are predicted from the model frame, while the macroblocks predicted from the previous frame or encoded in INTRA mode are grey. The mouth is not predicted from the model frame thus avoiding the prediction error coding from the generic teeth in Fig. 5.8. The same applies to macroblocks containing the object silhouette, since the background of the scene is not correctly modeled.
Figure 5.11: Frame 66 of the IJlumination sequence coded at about the same bit-rate using TMN-10, and the MAC coder without and with illumination compensation, a) TMN-10 (33.1 dB PSNR, 6764 bits), b) MAC without illumination compensation (33.8 dB PSNR, 5239 bits), c) MAC with illumination compensation (35.1 dB PSNR, 5160 bits).
Figure 5.11: Frame 66 of the IJlumination sequence coded at about the same bit-rate using TMN-10, and the MAC coder without and with illumination compensation, a) TMN-10 (33.1 dB PSNR, 6764 bits), b) MAC without illumination compensation (33.8 dB PSNR, 5239 bits), c) MAC with illumination compensation (35.1 dB PSNR, 5160 bits).
Figure 5.12: Temporal evolution of PSNR and bit-rate for the sequence Clapper Board. Both coders use a DCT quantizer parameter of 31; a) PSNR, b) bit-rate.
Figure 5.12: Temporal evolution of PSNR and bit-rate for the sequence Clapper Board. Both coders use a DCT quantizer parameter of 31; a) PSNR, b) bit-rate.
Figure 5.13: Rate-distortion plot for sequence Clapper Board; a) PSNR averaged over the entire sequence, b) PSNR measured for the first 60 frames that show the clapper board.
Figure 5.13: Rate-distortion plot for sequence Clapper Board; a) PSNR averaged over the entire sequence, b) PSNR measured for the first 60 frames that show the clapper board.
Figure 5.14: Frame 54 of the sequence Clapper Board. a) Model frame with clapper board missing; b) MAC, 36.1 dB, 2900 bits; c) TMN-10 with same bit-rate as MAC, 32.7 dB, 3200 bits; d) TMN-10 with same PSNR as MAC, 36.0 dB, 5800 bits.
Figure 5.14: Frame 54 of the sequence Clapper Board. a) Model frame with clapper board missing; b) MAC, 36.1 dB, 2900 bits; c) TMN-10 with same bit-rate as MAC, 32.7 dB, 3200 bits; d) TMN-10 with same PSNR as MAC, 36.0 dB, 5800 bits.
Figure A.1: Scene model and coordinate systems. The virtual scene that approximates the real world consists of multiple objects, light sources, and a camera as shown in Fig. A.1. The objects in the scene can either be rigid
Figure A.1: Scene model and coordinate systems. The virtual scene that approximates the real world consists of multiple objects, light sources, and a camera as shown in Fig. A.1. The objects in the scene can either be rigid
Figure A.2: Components of the camera model. A.3.1 Perspective Projection
Figure A.2: Components of the camera model. A.3.1 Perspective Projection
Figure A.3: Perspective projection and coordinate systems. TH84, Tek95].
Figure A.3: Perspective projection and coordinate systems. TH84, Tek95].
of both global and local motion. In the following table, the facial animation parameters of the head model described in Section 3.4.1 are listed. The names of the FAPs correspond directly to the parameters defined in MPI translational g of both global EG-4 |[MPG99] but the actual parameter numbers do not. In addition, 3 obal head movement parameters are added to obtain a uniform description and local motion.
of both global and local motion. In the following table, the facial animation parameters of the head model described in Section 3.4.1 are listed. The names of the FAPs correspond directly to the parameters defined in MPI translational g of both global EG-4 |[MPG99] but the actual parameter numbers do not. In addition, 3 obal head movement parameters are added to obtain a uniform description and local motion.
B. Facial Animation Parameters
B. Facial Animation Parameters
Table C.2: Absolute error of the estimated motion parameters averaged over 100 frames. The frames of the sequence are rendered from a 3-D model with distorted control point positions. All errors are measured in millimeters except for parameters 43-45 which represent angular errors given in degrees. C. Numerical Estimation Results
Table C.2: Absolute error of the estimated motion parameters averaged over 100 frames. The frames of the sequence are rendered from a 3-D model with distorted control point positions. All errors are measured in millimeters except for parameters 43-45 which represent angular errors given in degrees. C. Numerical Estimation Results
Institute, University of Erlangen-Nuremberg
Institute, University of Erlangen-Nuremberg
Related topics:
Video Coding
580 California St., Suite 400
San Francisco, CA, 94104
© 2025 Academia. All rights reserved