Academia.eduAcademia.edu

Figure 11 – uploaded by Chen Chen

See full PDFdownloadDownload figure
The flowchart of the real-time operation of the system is shown in Fig. 11. The detection of an action start and end i: continuously performed. After detecting an action start, the fusion classification method is activated while monitoring fo1 the action end. Note that the DMM gets computed frame by frame. The DMM feature computation is completed when the end of an action is detected. Fig. 11. Flowchart of the real-time human action recognition system.

Figure 11 The flowchart of the real-time operation of the system is shown in Fig. 11. The detection of an action start and end i: continuously performed. After detecting an action start, the fusion classification method is activated while monitoring fo1 the action end. Note that the DMM gets computed frame by frame. The DMM feature computation is completed when the end of an action is detected. Fig. 11. Flowchart of the real-time human action recognition system.

Related Figures (18)

Fig. 1. Microsoft Kinect depth sensor. Kinect is a low-cost RGB-Depth camera sensor introduced by Microsoft for human-computer interface applications. It comprises a color camera, an infrared (IR) emitter, an IR depth sensor, a tilt motor, a microphone array, and an LED light. A picture of the Kinect sensor or depth camera is shown in Fig. 1. This sensor can capture 16-bit depth images with a resolution of 320x240 pixels. Two example depth images are depicted in Fig. 2. The frame rate is approximately 30 frames per second. In addition, the Kinect SDK [14] is a publicly available software package which can be used to track 20 body skeleton joints (see Fig. 3) and their 3D spatial positions.
Fig. 1. Microsoft Kinect depth sensor. Kinect is a low-cost RGB-Depth camera sensor introduced by Microsoft for human-computer interface applications. It comprises a color camera, an infrared (IR) emitter, an IR depth sensor, a tilt motor, a microphone array, and an LED light. A picture of the Kinect sensor or depth camera is shown in Fig. 1. This sensor can capture 16-bit depth images with a resolution of 320x240 pixels. Two example depth images are depicted in Fig. 2. The frame rate is approximately 30 frames per second. In addition, the Kinect SDK [14] is a publicly available software package which can be used to track 20 body skeleton joints (see Fig. 3) and their 3D spatial positions.
Fig. 2. Example depth images from Kinect depth sensor.
Fig. 2. Example depth images from Kinect depth sensor.
Fig. 3. Skeleton joints provided by Kinect depth sensor.
Fig. 3. Skeleton joints provided by Kinect depth sensor.
The wearable inertial sensor used in this work is a small size (1”x1.5”) wireless inertial sensor built in the Embedded Signal Processing (ESP) Laboratory at Texas A&M University [15]. This sensor captures 3-axis acceleration, 3-axis angular velocity and 3-axis magnetic strength, which are transmitted wirelessly via a Bluetooth link to a laptop/PC. This wearable inertial sensor is shown in Fig. 4. The sampling rate of the sensor is 50 Hz and its measuring range is +8g for acceleration and +1000 degrees/second for rotation. It is worth mentioning that other commercially available inertial sensors can also be used in place of this inertial sensor. For practicality reasons or to avoid the intrusiveness associated with asking subjects to wear multiple inertial sensors, only one inertial sensor is considered in our work, either worn on the right wrist (similar to a watch) or the right thigh as depicted in Fig. 5 depending on the action of interest to be recognized in a particular application More explanations about the placement of the sensor for different actions are stated in Section IV. Fig. 6 shows the inertial sensor signals (3-axis accelerations and 3-axis angular velocities) for the action right hand wave. Fig. 4. Wearable inertial sensor developed in the ESP Lab.
The wearable inertial sensor used in this work is a small size (1”x1.5”) wireless inertial sensor built in the Embedded Signal Processing (ESP) Laboratory at Texas A&M University [15]. This sensor captures 3-axis acceleration, 3-axis angular velocity and 3-axis magnetic strength, which are transmitted wirelessly via a Bluetooth link to a laptop/PC. This wearable inertial sensor is shown in Fig. 4. The sampling rate of the sensor is 50 Hz and its measuring range is +8g for acceleration and +1000 degrees/second for rotation. It is worth mentioning that other commercially available inertial sensors can also be used in place of this inertial sensor. For practicality reasons or to avoid the intrusiveness associated with asking subjects to wear multiple inertial sensors, only one inertial sensor is considered in our work, either worn on the right wrist (similar to a watch) or the right thigh as depicted in Fig. 5 depending on the action of interest to be recognized in a particular application More explanations about the placement of the sensor for different actions are stated in Section IV. Fig. 6 shows the inertial sensor signals (3-axis accelerations and 3-axis angular velocities) for the action right hand wave. Fig. 4. Wearable inertial sensor developed in the ESP Lab.
Fig. 5. Inertial sensor placements: right wrist or right thigh. Fig. 6. Inertial sensor signals (3-axis accelerations and 3-axis angular velocities) for the action right hand wave.
Fig. 5. Inertial sensor placements: right wrist or right thigh. Fig. 6. Inertial sensor signals (3-axis accelerations and 3-axis angular velocities) for the action right hand wave.
Fig. 7. DMMs generated from a sample video of the action one hand wave.
Fig. 7. DMMs generated from a sample video of the action one hand wave.
Fig. 8. Action segmentation illustration using skeleton joint positions. d =a (x—a,)? + —y,)? + @— 2). (6) If for m, consecutive skeleton frames, all the distances are greater than a specified sensitivity og, the start of an action is triggered. If for m, consecutive skeleton frames, all the distances are less than or equal to the specified sensitivity o4, the end of an action is triggered. Fig. 8 illustrates the procedure of using skeleton joint positions to indicate the start and end of an action. The use of m, consecutive skeleton frames avoids responding to possible signal jitters.
Fig. 8. Action segmentation illustration using skeleton joint positions. d =a (x—a,)? + —y,)? + @— 2). (6) If for m, consecutive skeleton frames, all the distances are greater than a specified sensitivity og, the start of an action is triggered. If for m, consecutive skeleton frames, all the distances are less than or equal to the specified sensitivity o4, the end of an action is triggered. Fig. 8 illustrates the procedure of using skeleton joint positions to indicate the start and end of an action. The use of m, consecutive skeleton frames avoids responding to possible signal jitters.
Fig. 12. Sample shots of the 27 actions in the UTD-MHAD database. corrupted sequences, the dataset includes 861 data sequences. Sample shots of the 27 actions in the UTD-MHAD are shown in Fig. 12. The wearable inertial sensor was placed on the subjects’ right wrists for actions (1) through (21) which were hand type movements, and on the subjects’ right thigh for actions (22) through (27) which were leg type movements.
Fig. 12. Sample shots of the 27 actions in the UTD-MHAD database. corrupted sequences, the dataset includes 861 data sequences. Sample shots of the 27 actions in the UTD-MHAD are shown in Fig. 12. The wearable inertial sensor was placed on the subjects’ right wrists for actions (1) through (21) which were hand type movements, and on the subjects’ right thigh for actions (22) through (27) which were leg type movements.
ig. 13. Classification performance (recognition rates per action class and overall recognition rate) when using Kinect sensor only, inertial sensor nly, and Kinect and inertial sensors fusion for the subject-generic experiment.
ig. 13. Classification performance (recognition rates per action class and overall recognition rate) when using Kinect sensor only, inertial sensor nly, and Kinect and inertial sensors fusion for the subject-generic experiment.
Fig. 14. Classification performance (recognition rates per action class and overall recognition rate) when using Kinect sensor only, inertial sensor only, and Kinect and inertial sensors fusion for the subject-specific experiment.
Fig. 14. Classification performance (recognition rates per action class and overall recognition rate) when using Kinect sensor only, inertial sensor only, and Kinect and inertial sensors fusion for the subject-specific experiment.
Fig. 15. A misclassification case using Kinect and inertial sensor fusion.
Fig. 15. A misclassification case using Kinect and inertial sensor fusion.
achieved higher overall classification performance compared to using each sensor individually. Fig. 16. Recognition confusion matrix when using Kinect sensor only.
achieved higher overall classification performance compared to using each sensor individually. Fig. 16. Recognition confusion matrix when using Kinect sensor only.
Fig. 17. Recognition confusion matrix when using inertial sensor only. Fig. 18. Recognition confusion matrix when using Kinect and inertial sensors fusion.
Fig. 17. Recognition confusion matrix when using inertial sensor only. Fig. 18. Recognition confusion matrix when using Kinect and inertial sensors fusion.
ROCESSING TIMES TAKEN BY THE MAJOR COMPONENTS OF THE REAL-TIME SYSTEM Fig. 19. Real-time recognition confusion matrix.
ROCESSING TIMES TAKEN BY THE MAJOR COMPONENTS OF THE REAL-TIME SYSTEM Fig. 19. Real-time recognition confusion matrix.
Related topics:
Wearable ComputingWearable sensorsActivity RecognitionHuman Activity RecognitionAction RecognitionSensor Data FusionSensor FusionMultimodal FusionMulti Sensor Data FusionHuman Action Recognition
580 California St., Suite 400
San Francisco, CA, 94104
© 2025 Academia. All rights reserved