3D human pose estimation using 2D body part detectors

2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition

Accurate 3D human pose estimation from single images is possible with sophisticated deep-net architectures that have been trained on very large datasets. However, this still leaves open the problem of capturing motions for which no such database exists. Manual annotation is tedious, slow, and error-prone. In this paper, we propose to replace most of the annotations by the use of multiple views, at training time only. Specifically, we train the system to predict the same pose in all views. Such a consistency constraint is necessary but not sufficient to predict accurate poses. We therefore complement it with a supervised loss aiming to predict the correct pose in a small set of labeled images, and with a regularization term that penalizes drift from initial predictions. Furthermore, we propose a method to estimate camera pose jointly with human pose, which lets us utilize multiview footage where calibration is difficult, e.g., for pan-tilt or moving handheld cameras. We demonstrate the effectiveness of our approach on established benchmarks, as well as on a new Ski dataset with rotating cameras and expert ski motion, for which annotations are truly hard to obtain.

2019 16th International Conference on Machine Vision Applications (MVA), 2019

We propose a novel hybrid human 3D body pose estimation method that uses RGBD input. The method relies on a deep neural network to get an initial 2D body pose. Using depth information from the sensor, a set of 2D landmarks on the body are transformed in 3D. Then, a multiple hypothesis tracker uses the obtained 2D and 3D body landmarks to estimate the 3D body pose. In order to safeguard from observation errors, each human pose hypothesis considered by the tracker is constructed using a gradient descent optimization scheme that is applied to a subset of the body landmarks. Landmark selection is driven by a set of geometric constraints and temporal continuity criteria. The resulting 3D poses are evaluated by an objective function that calculates densely the discrepancy between the 3D structure of the rendered 3D human body model and the actual depth observed by the sensor. The quantitative experiments show the advantages of the proposed method over a baseline that directly uses all landmark observations for the optimization, as well as over other recent 3D human pose estimation approaches.

2014

We advocate the inference of qualitative information about 3D human pose, called posebits, from images. Posebits represent boolean geometric relationships between body parts (e.g. left-leg in front of right-leg or hands close to each other). The advantages of posebits as a mid-level representation are 1) for many tasks of interest, such qualitative pose information may be sufficient (e.g. semantic image retrieval), 2) it is relatively easy to annotate large image corpora with posebits, as it simply requires answers to yes/no questions; and 3) they help resolve challenging pose ambiguities and therefore facilitate the difficult talk of image-based 3D pose estimation. We introduce posebits, a posebit database, a method for selecting useful posebits for pose estimation and a structural SVM model for posebit inference. Experiments show the use of posebits for semantic image retrieval and for improving 3D pose estimation.

2013 IEEE Conference on Computer Vision and Pattern Recognition, 2013

In this work, we address the problem of estimating 2d human pose from still images. Recent methods that rely on discriminatively trained deformable parts organized in a tree model have shown to be very successful in solving this task. Within such a pictorial structure framework, we address the problem of obtaining good part templates by proposing novel, non-linear joint regressors. In particular, we employ two-layered random forests as joint regressors. The first layer acts as a discriminative, independent body part classifier. The second layer takes the estimated class distributions of the first one into account and is thereby able to predict joint locations by modeling the interdependence and co-occurrence of the parts. This results in a pose estimation framework that takes dependencies between body parts already for joint localization into account and is thus able to circumvent typical ambiguities of tree structures, such as for legs and arms. In the experiments, we demonstrate that our body parts dependent joint regressors achieve a higher joint localization accuracy than tree-based state-of-the-art methods.

IET Computer Vision, 2017

This work targets 2D articulated human pose estimation (i.e. localization of body limbs) in stereo videos. Although in recent years depth-based devices (e.g. Microsoft Kinect) have gained popularity, as they perform very well in controlled indoor environments (e.g. living rooms, operating theatres or gyms), they suffer clear problems in outdoor scenarios and, therefore, human pose estimation is still an interesting unsolved problem. We propose here a novel approach that is able to localize upper-body keypoints (i.e. shoulders, elbows, wrists) in temporal sequences of stereo image pairs. Our method starts by locating and segmenting people in the image pairs by using disparity and appearance information. Then, a set of candidate body poses is computed for each view independently. Finally, temporal and stereo consistency is applied to estimate a final 2D pose. We validate our model on three challenging datasets: "Stereo Human Pose Estimation Dataset", "Poses in the Wild" and "INRIA 3DMovie". The experimental results show that our model not only establishes new state-of-the-art results on stereo sequences, but also brings improvements in monocular sequences.

Research Square (Research Square), 2024

This paper presents a comprehensive survey and methodology for deep learningbased solutions in articulated human pose estimation (HPE). Recent advances in deep learning have revolutionized the HPE field, with the capturing system transitioning from multi-modal to a regular color camera and from multi-views to a monocular view, opening up numerous applications. However, the increasing variety of deep network architectures has resulted in a vast literature on the topic, making it challenging to identify commonalities and differences among 1 diverse HPE approaches. Therefore, this paper serves two objectives: firstly, it provides a thorough survey of over 100 research papers published since 2015, focusing on deep learning-based solutions for monocular HPE; secondly, it develops a comprehensive methodology that systematically combines existing works and summarizes a unified framework for the HPE problem and its modular components. Unlike previous surveys, this study places emphasis on methodology development in order to provide betters insights and learning opportunities for researchers in the field of computer vision. The paper also summarizes and discusses the quantitative performance of the reviewed methods on popular datasets, while highlighting the challenges involved, such as occlusion and viewpoint variation. Finally, future research directions, such as incorporating temporal information and 3D pose estimation, along with potential solutions to address the remaining challenges in HPE, are presented.

IEEE Transactions on Pattern Analysis and Machine Intelligence, 2006

We describe a learning based method for recovering 3D human body pose from single images and monocular image sequences. Our approach requires neither an explicit body model nor prior labelling of body parts in the image. Instead, it recovers pose by direct nonlinear regression against shape descriptor vectors extracted automatically from image silhouettes. For robustness against local silhouette segmentation errors, silhouette shape is encoded by histogram-of-shape-contexts descriptors. We evaluate several different regression methods: ridge regression, Relevance Vector Machine (RVM) regression and Support Vector Machine (SVM) regression over both linear and kernel bases. The RVMs provide much sparser regressors without compromising performance, and kernel bases give a small but worthwhile improvement in performance. Loss of depth and limb labelling information often makes the recovery of 3D pose from single silhouettes ambiguous. We propose two solutions to this: the first embeds the method in a tracking framework, using dynamics from the previous state estimate to disambiguate the pose; the second uses a mixture of regressors framework to return multiple solutions for each silhouette. We show that the resulting system tracks long sequences stably, and is also capable of accurately reconstructing 3D human pose from single images, giving multiple possible solutions in ambiguous cases. For realism and good generalization over a wide range of viewpoints, we train the regressors on images resynthesized from real human motion capture data. The method is demonstrated on a 54-parameter full body pose model, both quantitatively on independent but similar test data, and qualitatively on real image sequences. Mean angular errors of 4-5 degrees are obtained -a factor of 3 better than the current state of the art for the much simpler upper body problem.

SN Computer Science

Human pose estimation is the process of detecting the body keypoints of a person and can be used to classify different poses. Many researchers have proposed various ways to get a perfect 2D as well as a 3D human pose estimator that could be applied for various types of applications. This paper is a review of all the state-of-the-art architectures based on human pose estimation, the papers referred were based on the types of computer vision and machine learning algorithms, such as feed-forward neural networks, convolutional neural networks (CNN), OpenPose, MediaPipe, and many more. These different approaches are compared on various parameters, like the type of dataset used, the evaluation metric, etc. Different human pose datasets, such as COCO and MPII activity datasets with keypoints, as well as specific application-based datasets, are reviewed in this survey paper. Researchers may use these architectures and datasets in a range of domains, which are also discussed. The paper analyzes several approaches and architectures that can be used as a guide for other researchers to assist them in developing better techniques to achieve high accuracy.

We describe the first method to automatically estimate the 3D pose of the human body as well as its 3D shape from a single un-constrained image. We estimate a full 3D mesh and show that 2D joints alone carry a surprising amount of information about body shape. The problem is challenging because of the complexity of the human body, articulation, occlusion, clothing, lighting, and the inherent ambiguity in inferring 3D from 2D. To solve this, we first use a recently published CNN-based method, DeepCut, to predict (bottom-up) the 2D body joint locations. We then fit (top-down) a recently published statistical body shape model, called SMPL, to the 2D joints. We do so by minimizing an objective function that penalizes the error between the projected 3D model joints and detected 2D joints. Because SMPL captures correlations in human shape across the population, we are able to robustly fit it to very little data. We further leverage the 3D model to prevent solutions that cause interpenetration. We evaluate our method, SMPLify, on the Leeds Sports, HumanEva, and Human3.6M datasets, showing superior pose accuracy with respect to the state of the art.

2016

Until recently Intelligence, Surveillance, and Reconnaissance (ISR) focused on acquiring behavioral information of the targets and their activities. Continuous evolution of intelligence being gathered of the human centric activities has put increased focus on the humans, especially inferring their innate characteristics-size, shapes and physiology. These biosignatures extracted from the surveillance sensors can be used to deduce age, ethnicity, gender and actions, and further characterize human actions in unseen scenarios. However, recovery of pose and shape of humans in such monocular videos is inherently an ill-posed problem, marked by frequent depth and view based ambiguities due to self-occlusion, foreshortening and misalignment. The likelihood function often yields a highly multimodal posterior that is difficult to propagate even using the most advanced particle filtering(PF) algorithms. Motivated by the recent success of the discriminative approaches to efficiently predict 3D poses directly from the 2D images, we present several principled approaches to integrate predictive cues using learned regression models to sustain multimodality of the posterior during tracking. Additionally, these learned priors can be actively adapted to the test data using a likelihood based feedback mechanism. Estimated 3D poses are then used to fit 3D human shape model to each frame independently for inferring anthropometric biosignatures. The proposed system is fully automated, robust to noisy test data and has ability to swiftly recover from tracking failures even after confronting with significant errors. We evaluate the system on a large number of monocular human motion sequences.