Academia.eduAcademia.edu

Figure 5 – uploaded by Vasu Parameswaran

See full PDFdownloadDownload figure
the phases of an action can be quite difficult. Consid- ering walking once again, the left/right arms and legs trace four different planes and on each plane, there are atmost three joints we can identify at each phase - shoulder-elbow-hand and hip-knee-foot. We solve this problem by first identifying a range of frames delin- eated by alternate occurrences of the canonical pose C, (see Fig. 4). Two joints (say the right-hip and right- foot) for the start and end frames give us four points. The two invariants formed by a fifth moving point (say the right-knee) will trace two trajectories which we call invariance space trajectories (ISTs). We thus obtain the five points necessary for computing two invariants at each phase and still need to identify only three joints in each phase. The use of ISTs is advantageous for another important aspect: Recall from our continuity analysis that the closer the joints are to a plane, the more accurate is the representation. The overall dis- tances between the points for which we calculate in- variants in an IST, are larger than distances between points in a single frame and so the planarity of the five points in an IST is better. Other nice properties that ISTs buy us are independence from the frame rate and speed of the action, because the starting and ending instants (and hence, duration) of the ISTs are not fixed upfront. Rather, they are event driven (i.e. determined by the occurrence of specific canonical poses). Note that ISTs can be suitably employed for other actions end wan swereereeeeee coin oie: seceuiee aie

Figure 4 the phases of an action can be quite difficult. Consid- ering walking once again, the left/right arms and legs trace four different planes and on each plane, there are atmost three joints we can identify at each phase - shoulder-elbow-hand and hip-knee-foot. We solve this problem by first identifying a range of frames delin- eated by alternate occurrences of the canonical pose C, (see Fig. 4). Two joints (say the right-hip and right- foot) for the start and end frames give us four points. The two invariants formed by a fifth moving point (say the right-knee) will trace two trajectories which we call invariance space trajectories (ISTs). We thus obtain the five points necessary for computing two invariants at each phase and still need to identify only three joints in each phase. The use of ISTs is advantageous for another important aspect: Recall from our continuity analysis that the closer the joints are to a plane, the more accurate is the representation. The overall dis- tances between the points for which we calculate in- variants in an IST, are larger than distances between points in a single frame and so the planarity of the five points in an IST is better. Other nice properties that ISTs buy us are independence from the frame rate and speed of the action, because the starting and ending instants (and hence, duration) of the ISTs are not fixed upfront. Rather, they are event driven (i.e. determined by the occurrence of specific canonical poses). Note that ISTs can be suitably employed for other actions end wan swereereeeeee coin oie: seceuiee aie

Related Figures (28)

Figure 1. C, planarity. A human action can be thought of in terms of a starting pose P;, an ending pose P,, and a sequence of continuous transitions that take the body from pose ?; at time t = 0 to pose P, at time tf = T. We can eliminate rate variations by scaling time such that the action oc- curs from ¢f = 0 to t= 1. We can think of a phase of an action, ¢ that takes on a value in the interval [0, 1]. A phase value maps to a body pose P(t). An action then becomes the function P(f), t € [0,1]. We can represent the action in a view independent way by choosing key poses such that atleast five body joints align themselves approximately in a plane (we say approximate, here be- cause we can show (as we do in Section 3.3) that we can achieve good results even with approximate rather than total planarity). We call such poses canonical poses. This is indeed possible for a variety of human actions. Let us consider the example of the action walking. During walking, the hands, feet and head of the person fall in a vertical plane periodically (e.g. first image in Fig. 2). Figure 1 shows the distances of the right-foot and the head to the plane formed by the right-shoulder, eft-shoulder and the left-foot of a walking person from motion-capture data. The distances have been
Figure 1. C, planarity. A human action can be thought of in terms of a starting pose P;, an ending pose P,, and a sequence of continuous transitions that take the body from pose ?; at time t = 0 to pose P, at time tf = T. We can eliminate rate variations by scaling time such that the action oc- curs from ¢f = 0 to t= 1. We can think of a phase of an action, ¢ that takes on a value in the interval [0, 1]. A phase value maps to a body pose P(t). An action then becomes the function P(f), t € [0,1]. We can represent the action in a view independent way by choosing key poses such that atleast five body joints align themselves approximately in a plane (we say approximate, here be- cause we can show (as we do in Section 3.3) that we can achieve good results even with approximate rather than total planarity). We call such poses canonical poses. This is indeed possible for a variety of human actions. Let us consider the example of the action walking. During walking, the hands, feet and head of the person fall in a vertical plane periodically (e.g. first image in Fig. 2). Figure 1 shows the distances of the right-foot and the head to the plane formed by the right-shoulder, eft-shoulder and the left-foot of a walking person from motion-capture data. The distances have been
Figure 2. Canonical poses C1, C2 and C3. normalized by the head-to-toe length. A zero-crossing of the two curves indicates that the five joints are on a plane. Let us denote this canonical pose for walking, where the hands, feet and head lie approximately on a plane as C,. The detection of C by itself provides us a strong cue that a “walk-like” action is taking place and can become part of the action model for walking. In ad- dition to the C; pose, each corresponding leg and hand of the body trace areas lying approximately on a plane. In other words, the left hand and leg as well as the right hand and leg lie approximately on a plane in each walk cycle. Further, there exist two poses where the body limbs swing to their maximal positions and the end ef- fectors attain their maximum distances from the torso. Call these canonical poses Cz and C3 (see Fig. 2). There exist many joint combinations that could be used to represent C,, C2 and C3. The requirement to be satisfied of a joint combination is that the joints lie approximately on a plane for the corresponding pose. We can pick tuples of five such joints and use (1) to pre-compute a pair of invariants for each tu- ple. Pairs of invariants for several five-tuples can be used to represent a particular body-pose which oc- curs at a single instant of an action. For the action walking, a simple action model would thus be the re- peating sequence C, C2, C,, C3. Clearly this type of abstraction of an action in terms of invariants from a planar decomposition is possible only if we can lo- cate a significant number of poses in the action where five or more joints lie approximately on a plane and is not a completely general method for representing any arbitrary action. However, in practice, this is possi- ble for many other actions including jumping, waving, running, sitting-down etc. Indeed as Hoffman points out in his book Visual Intelligence (Hoffman, 1998), the tendency of the human body joints is to move in planes because of the manner in which their support- ing bones are structured and our tendency to minimize
Figure 2. Canonical poses C1, C2 and C3. normalized by the head-to-toe length. A zero-crossing of the two curves indicates that the five joints are on a plane. Let us denote this canonical pose for walking, where the hands, feet and head lie approximately on a plane as C,. The detection of C by itself provides us a strong cue that a “walk-like” action is taking place and can become part of the action model for walking. In ad- dition to the C; pose, each corresponding leg and hand of the body trace areas lying approximately on a plane. In other words, the left hand and leg as well as the right hand and leg lie approximately on a plane in each walk cycle. Further, there exist two poses where the body limbs swing to their maximal positions and the end ef- fectors attain their maximum distances from the torso. Call these canonical poses Cz and C3 (see Fig. 2). There exist many joint combinations that could be used to represent C,, C2 and C3. The requirement to be satisfied of a joint combination is that the joints lie approximately on a plane for the corresponding pose. We can pick tuples of five such joints and use (1) to pre-compute a pair of invariants for each tu- ple. Pairs of invariants for several five-tuples can be used to represent a particular body-pose which oc- curs at a single instant of an action. For the action walking, a simple action model would thus be the re- peating sequence C, C2, C,, C3. Clearly this type of abstraction of an action in terms of invariants from a planar decomposition is possible only if we can lo- cate a significant number of poses in the action where five or more joints lie approximately on a plane and is not a completely general method for representing any arbitrary action. However, in practice, this is possi- ble for many other actions including jumping, waving, running, sitting-down etc. Indeed as Hoffman points out in his book Visual Intelligence (Hoffman, 1998), the tendency of the human body joints is to move in planes because of the manner in which their support- ing bones are structured and our tendency to minimize
Figure 3. Euclidean norm of invariant distances for C,. Hence, if the invariant value is finite to begin with, we see that it varies infinitesimally for small deviations from a plane. This result justifies the use of thresh- olding for determining if the body is in a particular canonical pose. However, this is aqualified justifica- tion because although infinitesimal, the deviations in the invariant values (AJ, AJ,) are dependent upon the viewpoint T. In the extreme case, if the camera is very close to the subject, the denominator a will be small and the deviations will be large. However, for moderately distant camera positions, we are able to get reasonably good results with only a single threshold for all viewpoints. Figure 3 shows the Euclidean distances between the walking C, invariants and the area-cross- ratios against each frame for a walking subject. The
Figure 3. Euclidean norm of invariant distances for C,. Hence, if the invariant value is finite to begin with, we see that it varies infinitesimally for small deviations from a plane. This result justifies the use of thresh- olding for determining if the body is in a particular canonical pose. However, this is aqualified justifica- tion because although infinitesimal, the deviations in the invariant values (AJ, AJ,) are dependent upon the viewpoint T. In the extreme case, if the camera is very close to the subject, the denominator a will be small and the deviations will be large. However, for moderately distant camera positions, we are able to get reasonably good results with only a single threshold for all viewpoints. Figure 3 shows the Euclidean distances between the walking C, invariants and the area-cross- ratios against each frame for a walking subject. The
Figure 6. | Walk-cycle IST, same action, same viewpoint, five sub- jects. for all k in the window: Figure 5. | Walk-cycle IST, same subject, same viewpoint, five instances. Let F be a sequence of frames containing area-cross- ratios which we would like to match with P the set of IST invariants. Let f; ¢ F and p; € P where i=1..N, j=1..M. In other words, N is the number of frames and M is the number of phases. We seek the optimal map- ping w: P—F such that the cumulative distance be- tween the area cross-ratios of each frame with the phase that it is mapped to, namely ye d(p;, W(pj)) is min- imized. We have the boundary condition #(p1) = fi and (pm) = fn because the ISTs are delineated by canonical poses as explained earlier and the mappings of the ends are fixed.
Figure 6. | Walk-cycle IST, same action, same viewpoint, five sub- jects. for all k in the window: Figure 5. | Walk-cycle IST, same subject, same viewpoint, five instances. Let F be a sequence of frames containing area-cross- ratios which we would like to match with P the set of IST invariants. Let f; ¢ F and p; € P where i=1..N, j=1..M. In other words, N is the number of frames and M is the number of phases. We seek the optimal map- ping w: P—F such that the cumulative distance be- tween the area cross-ratios of each frame with the phase that it is mapped to, namely ye d(p;, W(pj)) is min- imized. We have the boundary condition #(p1) = fi and (pm) = fn because the ISTs are delineated by canonical poses as explained earlier and the mappings of the ends are fixed.
where
where
Figure 7. Body model. consisted of the joint angles of the action. This way, we were able to mix subjects and actions from different sources creating more motion capture sequences. The body model we employed consisted of fifteen joints (see Fig. 7) namely the head, hip, chest, shoulders, elbows, hands, hips, knees and feet.
Figure 7. Body model. consisted of the joint angles of the action. This way, we were able to mix subjects and actions from different sources creating more motion capture sequences. The body model we employed consisted of fifteen joints (see Fig. 7) namely the head, hip, chest, shoulders, elbows, hands, hips, knees and feet.
Figure 8. Start pose for walk-cycle, run-cycle, sit-down and end-pose for sit-down. in preparation for sitting down and the ending pose where the subject is finally seated. For forward-jump, we define the start of the action to be the pose when the feet are stationary and just about to move upwards and the end of the action when the feet become station- ary again (see Fig. 9). Model building was done in an automatic way with manual supervision only to verify correctness of the model. For walking and running, a planarity assessment program was run through one in- stance of each subject (12 subjects total) giving frames where the joints of interest for C, were on a plane. n our case, specifically, these joints were the head, right-shoulder, left-shoulder, left-foot, and right-foot for both walking and running. An invariant calculation program was run on those specific frames where the joints were closest to being planar, and the average invariant values were obtained. These frames became delineators for the ISTs. Next, invariance space trajec- tories were extracted automatically for frames between these delineator frames, and median values of the in- variants for each intervening frame were obtained. We used two ISTs each to model walking and running. The specific joints used were the same for both the ac- tions and they were {left-hip, left-foot, left-knee} and {right-hip, right-foot, right-knee}. Our choice of using the leg joints for the ISTs for walking and running was motivated by the fact that the motion of the legs exhib- ited more planarity and less variability than the motion of the arms. The two ISTs along with the C, invariant values for each action completed our action-model for the walking and running actions (Fig. 10 shows the ISTs). For sit-down, a stationarity assessment program was run first only on the feet and then on the feet and hip. These helped us determine the delineator poses for the action. We used two ISTs for representing the sit-down action. The specific joints chosen were {left- foot, hip, head} and {right-foot, hip, head}. We found that the trajectories formed by the elbows and hands We built models for four actions: walk-cycle, run-cycle, sit-down and forward-jump. The walk-cycle and run- cycle actions are cyclical in that any particular phase of the actions can become the starting pose of the ac- tion. We arbitrarily define the starting (and ending) pose of the actions as that where the body is in the Cy pose and where the left foot just passes by the right foot (see Fig. 8). For sit-down, we define the starting pose of the action where the feet become stationary
Figure 8. Start pose for walk-cycle, run-cycle, sit-down and end-pose for sit-down. in preparation for sitting down and the ending pose where the subject is finally seated. For forward-jump, we define the start of the action to be the pose when the feet are stationary and just about to move upwards and the end of the action when the feet become station- ary again (see Fig. 9). Model building was done in an automatic way with manual supervision only to verify correctness of the model. For walking and running, a planarity assessment program was run through one in- stance of each subject (12 subjects total) giving frames where the joints of interest for C, were on a plane. n our case, specifically, these joints were the head, right-shoulder, left-shoulder, left-foot, and right-foot for both walking and running. An invariant calculation program was run on those specific frames where the joints were closest to being planar, and the average invariant values were obtained. These frames became delineators for the ISTs. Next, invariance space trajec- tories were extracted automatically for frames between these delineator frames, and median values of the in- variants for each intervening frame were obtained. We used two ISTs each to model walking and running. The specific joints used were the same for both the ac- tions and they were {left-hip, left-foot, left-knee} and {right-hip, right-foot, right-knee}. Our choice of using the leg joints for the ISTs for walking and running was motivated by the fact that the motion of the legs exhib- ited more planarity and less variability than the motion of the arms. The two ISTs along with the C, invariant values for each action completed our action-model for the walking and running actions (Fig. 10 shows the ISTs). For sit-down, a stationarity assessment program was run first only on the feet and then on the feet and hip. These helped us determine the delineator poses for the action. We used two ISTs for representing the sit-down action. The specific joints chosen were {left- foot, hip, head} and {right-foot, hip, head}. We found that the trajectories formed by the elbows and hands We built models for four actions: walk-cycle, run-cycle, sit-down and forward-jump. The walk-cycle and run- cycle actions are cyclical in that any particular phase of the actions can become the starting pose of the ac- tion. We arbitrarily define the starting (and ending) pose of the actions as that where the body is in the Cy pose and where the left foot just passes by the right foot (see Fig. 8). For sit-down, we define the starting pose of the action where the feet become stationary
Figure 11. IST for sit-down. Figure 10. _ Median ISTs for walk-cycle and run-cycle.
Figure 11. IST for sit-down. Figure 10. _ Median ISTs for walk-cycle and run-cycle.
eure 9. Forward-jump starting and ending poses, images from www.mocap.cs.umd.edu.
eure 9. Forward-jump starting and ending poses, images from www.mocap.cs.umd.edu.
Figure 13. Probability distribution of C; invariants for walking. Figure 12. IST for forward-jump.
Figure 13. Probability distribution of C; invariants for walking. Figure 12. IST for forward-jump.
Figure 15. Walk seq. 20, frame no. 100, all viewpoints. For the specific joint combinations that we used to model actions, the inter and intra-subject variability in the invariants was quite small. Figures 13 and 14 show histograms for the walking and running C| invariant values for 1745 walk-cycles and 1980 run-cycles re- spectively from 12 different subjects and 5 different viewpoints each. The variability was also an artifact resulting from inconsistencies in the placement of sen- sors on the motion capture subjects because these were obtained from different sources. For example from one source, the ‘shoulder’ sensor was called “upper arm’ and was placed slightly below the shoulder position. Similarly the ‘knee’ sensor was placed slightly below
Figure 15. Walk seq. 20, frame no. 100, all viewpoints. For the specific joint combinations that we used to model actions, the inter and intra-subject variability in the invariants was quite small. Figures 13 and 14 show histograms for the walking and running C| invariant values for 1745 walk-cycles and 1980 run-cycles re- spectively from 12 different subjects and 5 different viewpoints each. The variability was also an artifact resulting from inconsistencies in the placement of sen- sors on the motion capture subjects because these were obtained from different sources. For example from one source, the ‘shoulder’ sensor was called “upper arm’ and was placed slightly below the shoulder position. Similarly the ‘knee’ sensor was placed slightly below
Figure 16. Incorrect detection of C; pose for a walking sequence from viewpoint 3. We evaluated our algorithm on two modalities of in- put: arbitrary projections of motion capture data and manually segmented real image sequences. A set of unknown actions observed from different viewpoints and performed by different subjects at different speeds were input to the action recognition system. These ac- tions were continuous, meaning that each sequence had one or more instances of an action being performed, and the starting and ending times of the action were unknown (to the system). The system not only had to
Figure 16. Incorrect detection of C; pose for a walking sequence from viewpoint 3. We evaluated our algorithm on two modalities of in- put: arbitrary projections of motion capture data and manually segmented real image sequences. A set of unknown actions observed from different viewpoints and performed by different subjects at different speeds were input to the action recognition system. These ac- tions were continuous, meaning that each sequence had one or more instances of an action being performed, and the starting and ending times of the action were unknown (to the system). The system not only had to
Figure 17. Front-view walking sequence and C, invariant norm.
Figure 17. Front-view walking sequence and C, invariant norm.
Figure 18. Side-view walking sequence and C, invariant norm.
Figure 18. Side-view walking sequence and C, invariant norm.
these were detected as sit-down actions by the system, increasing the false alarm rate. Another problem was that many subjects were jogging rather than running for some of the sequences labeled running, and the algo- rithm in some of these cases classified these sequences incorrectly as walking. Table 2 shows the confusion matrices for all viewpoints combined, which shows these problems. Note that the confusion matrix corre- sponds to a total of 6000 expected actions (1200 x 5). Yet another particular problem with the approach was the performance when the person is relatively still as was the case at the end of all of the forward jump and sit-down actions. Still, there are small joint motions that trigger the algorithm to detect local minima at
these were detected as sit-down actions by the system, increasing the false alarm rate. Another problem was that many subjects were jogging rather than running for some of the sequences labeled running, and the algo- rithm in some of these cases classified these sequences incorrectly as walking. Table 2 shows the confusion matrices for all viewpoints combined, which shows these problems. Note that the confusion matrix corre- sponds to a total of 6000 expected actions (1200 x 5). Yet another particular problem with the approach was the performance when the person is relatively still as was the case at the end of all of the forward jump and sit-down actions. Still, there are small joint motions that trigger the algorithm to detect local minima at
Figure 20. Side-view running sequence and C| invariant norm. Figure 19. Top-view walking sequence and C invariant norm.
Figure 20. Side-view running sequence and C| invariant norm. Figure 19. Top-view walking sequence and C invariant norm.
Figure 21. Front-view running sequence and C| invariant norm.
Figure 21. Front-view running sequence and C| invariant norm.
Figure 22. Top-view running sequence and C, invariant norm.
Figure 22. Top-view running sequence and C, invariant norm.
containing one walk-cycle, shot from a front view along with the marked joints and skeleton. Beside the frame is a plot showing the distance between the area- cross-ratios and the invariants for the C; pose. Note that similar to the motion capture sequences, the invariant distance approaches zero thrice, reflecting the fact that the body assumes the Cy pose thrice in a walk-cycle. Similarly, Figs. 18 and 19 show a sample frame and C; invariant norms for a side view and top-view walking sequence. With regard to the top-view, although the curves exhibit several spurious local minima, the ISTs were able to eliminate all of them except the correct
containing one walk-cycle, shot from a front view along with the marked joints and skeleton. Beside the frame is a plot showing the distance between the area- cross-ratios and the invariants for the C; pose. Note that similar to the motion capture sequences, the invariant distance approaches zero thrice, reflecting the fact that the body assumes the Cy pose thrice in a walk-cycle. Similarly, Figs. 18 and 19 show a sample frame and C; invariant norms for a side view and top-view walking sequence. With regard to the top-view, although the curves exhibit several spurious local minima, the ISTs were able to eliminate all of them except the correct
Figure 24. Sample frames for forward-jump top-view, and observed and expected ISTs. gure 23. Sample frames for sit-down top-view, and observed and expected ISTs
Figure 24. Sample frames for forward-jump top-view, and observed and expected ISTs. gure 23. Sample frames for sit-down top-view, and observed and expected ISTs
Table 3. Real image sequences, 50 total actions. blocks one’s view below). Hence the “top” view for the real image sequence was slightly different than the top view for the motion capture sequences. The number of real image sequences used for the performance evalu- ation is not high enough to enable us to draw definite conclusions about the nature of the failures and the dif- ferences in performance between different viewpoints. Nevertheless, this demonstrates the applicability of the approach to real image sequences and it can be seen that the overall performance is quite good given the number of viewpoints, subjects and actions, different speeds of action and noisy input.
Table 3. Real image sequences, 50 total actions. blocks one’s view below). Hence the “top” view for the real image sequence was slightly different than the top view for the motion capture sequences. The number of real image sequences used for the performance evalu- ation is not high enough to enable us to draw definite conclusions about the nature of the failures and the dif- ferences in performance between different viewpoints. Nevertheless, this demonstrates the applicability of the approach to real image sequences and it can be seen that the overall performance is quite good given the number of viewpoints, subjects and actions, different speeds of action and noisy input.
Related topics:
Computer VisionHuman MotionAction RecognitionImage Sequence
580 California St., Suite 400
San Francisco, CA, 94104
© 2025 Academia. All rights reserved