2.1 Weakly and self-supervised approaches in 3D human pose estimation

A number of works are devoted to weakly and self-supervised methods for 3D human pose estimation. We also refer to papers [16–18], where authors studied how to use weakness and self-supervision for overcoming some difficulties connected with receipt of 3D ground truth labels. In a way, self-supervised and weakly supervised methods sketched in this paper provide a partial solution to this problem that the fully supervised approach [19–21] addressing direct estimation of 3D joint coordinates from images using regression networks. Pavlakos et al. [22] utilized a pictorial structure model to derive global pose configurations from key-point heat maps in multi-view images. However, their method required full camera calibration and a keypoint detector for generating 2D heat maps. Rhodin et al. [16] used multi-view consistency restraints to guide a network, minimizing the need for extensive 3D ground truth data to avoid pose collapse issues. Nonetheless, the scarcity of in-the-wild 3D ground truth data remains a limitation. Drover et al. [17] introduced an adversarial framework for weak supervision, learning 3D structures solely from random 2D projections. Notably, their method eliminated the need for correspondence between 2D and 3D points but still relied on 3D labels. Zhou et al. [23] integrated 2D and 3D data, applying a Euclidean loss for 3D data and proposing a weak supervision loss based on 2D annotations and prior knowledge of the human skeleton. Despite improving applicability to outdoor images, their approach required both 2D and 3D labels. Wandt et al. [24] pioneered a generative adversarial network with three components, including a pose and camera estimation network, a critic network, and a re-projection network. Their method was also accompanied by another feature: camera estimation network and re-projection layer for re-projection of the estimated 3D pose to 2D space.

Rhodin et al. [25] proposed a weakly supervised method to transfer the body representation from the 3D space that is adapted to the geometric analyses of the multiple-view images. In their Self-supervised learning paper, Kocabas et al. [26] introduced Epipolar-Pose which doesn’t require 3D ground truth data or cameras extrinsics information instead it uses epipolar geometry to produce annotations from key-points. A self-supervised method of 3D human pose estimation from monocular images was presented by Li et al. [27, 28]. Altogether, these works help the investigation of the methods that need little control and supervision, which respond to issues related to the annotated data availability. Extending the weak supervision topic further, Wu et al. [29] introduced a work on weakly supervised 3D human pose estimation using the transfer of 2D HPE annotation information from large scale RGB datasets to the 3D task. Roy et al. [30] discussed a supervised method making use of multi-view geometrical restraints which can be considered as a form of self-supervision where the labelled data is limited. Cha et al. [31] proposed a new self-supervised learning paradigm for 3D human pose and shape recovery in single shot 2D images. Altogether, these works are connected to the investigation of the self-supervised learning frameworks and present progress in handling supervision issues in 3D pose estimation.

3.1 TRIPOSE-NET

The first baseline network adopted in the TRIPOSE-NET module is ResNet-50 due to its ability to mitigate on vanishing gradients which is a common problem in deep convolutional network. The nature of these vanishing gradients is that they reduce the effectiveness of convolutional networks moving to the deeper layers while it seems that more layers should be better at capturing finer details. The possibility of degradation in performance especially in deep networks is also prevented by residual connections which enable the gradients to flow through the depth of the network thereby making the optimisation possible. Owing to its versatility across a plethora of tasks, the Residual Network (ResNet) is an optimal choice as the backbone for these pose estimation pipelines, providing liberty to extract abstract and densely intricate features by virtue of its deep and robust distinctive features. To elevate the performance of the ResNet-50 module, an SKNet Block is introduced at the end of the 3rd convolution block and at the tail-end of the feature extractor, i.e., just before the global average pooling layer.

This forms an effective feature extraction module named SK-ResNet 50, as illustrated in Fig 1. The addition of the SKNet block at the deeper layers, primarily the bottleneck blocks, assures that more global context is captured as the receptive field is larger in this region and that the layers have a high spatial resolution. Additionally, the placement of the SKNet block before the global average pooling allows for adaptive feature selection on the basis of global context prior to the final prediction. Furthermore, using SKNet in ResNet-50 augments the model’s capabilities in dealing with multi-scale features by acting as an auxiliary aid in selecting the appropriate kernel sizes for capturing both fine-grained and global details to achieve optimal performance in pose estimation.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Modified feature extraction block (ResNet-50) using selective kernel network.

https://doi.org/10.1371/journal.pone.0310831.g001

Initially, a deconvolution layer is used to project the extracted feature from the 2D pose estimation module as Heat-Maps denoted by M_h, where . Here (m, n) represent the dimensions of M_h and to acquire the 2D key-point positions (x, y) from the respective maps M_h for all nodes N such that , we apply an argmax function ν.

The resultant human poses A_{(i, j)} are in two-dimensional representation, whereas the acquisition of 3D poses requires supervision under a 3-dimensional paradigm. The annotation of 3D poses for training endeavors is both time-consuming and resource-intensive. Herein, we propose a weakly supervised method, utilizing the 2D poses A_{(i, j)} from initial training and epipolar geometry to triangulate the 3D correspondences. The volumetric key-point mappings given by are cached and taken as ground truths with pseudo labels to train the TRIPOSE-NET tailored to 3D pose estimations C_{(i, j, k)}, significantly curbing the need for pre-annotation of the data for 3D pose estimates.

3.2 Selective kernel network

Selective Kernel Network (SKNet) [32] emulates the dynamic modulation of receptive fields of visual-cortical neurons based on stimuli. This emulation allowed the extension of CNN with the integration of adaptive kernel sizing based on the input information. In SKNet, a Selective Kernel (SK) unit containing multiple branches with varying kernel sizes is fused in a way that ascertains the appropriate selection of the receptive field size, as visually explained in Fig 2. This adaptability equips the model with the ability to focus only on the most informative regions, akin to an attention mechanism. With proficient multi-scale feature integration owing to the dynamic allocation of receptive field size, SKNet extracts both coarse-level and fine-grained features imperative for effective pose estimation [33]. Additionally, the selective kernel feature provides a more expressive feature representation, with different parts of the network specializing in capturing diverse features. Relationships between features are meaningfully captured, resulting in improved pose estimation. A task where the preservation of the relationship between the joints is a significant facet. Within the proposed TRIPOSE-NET architecture, one of the SK units is added before the 4th convolution block of ResNet-50, and a second SK unit is added before the Global-Average-Pooling Layer, as shown in Fig 1. The 1st SK unit is dual-branched with two kernels of sizes 3x3 and 5x5. The 5x5 kernel is actually a dilated version of a 3x3 kernel with a dilation factor of 2. Dilated convolutions [34], as shown in Fig 3, help capture contextual information by introducing gaps within the elements of the filter, effectively reducing the parameter count and enhancing the network’s global awareness. The 2nd SK unit consists of three branches with kernel sizes of 3x3, 5x5, and 7x7. The first kernel is without dilation, while the other two are 3x3 kernels dilated with factors 2 and 3.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Layer topology of selective kernel unit.

https://doi.org/10.1371/journal.pone.0310831.g002

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Visual illustration of dilated convolution kernel and its receptive field.

https://doi.org/10.1371/journal.pone.0310831.g003

The SK unit is segregated into three operations, namely Split, Fuse, and Select. Within the Split operation, the feature map F_m is propagated to the SK module, where two transformations and are performed. Each transformation involves a depth-wise convolution, followed by batch normalization, and finally a ReLU activation, in this sequence. For the first integration of the SK block within the ResNet-50 module, a dual-branch transformation and is used with kernel sizes 3x3 and 5x5, respectively. While the second embedding of the SK unit is based on a three-branched transformation , , and , each branch with a varying receptive field size as shown in Fig 2. The respective transformations within the Split operation produce outputs , , , where n represents the number of transformations.

The Fuse operation involves the element-wise summation of each transformation, which in our case involves both the two-branched U₁ and three-branched U₂ transformations, given by: (1) (2)

The outputs U₁ and U₂ from the fused transformations are passed through a Global Average Pooling Operation to obtain channel-wise global information statistics. The Pooling Operation G_p is applied across the spatial dimensions W × H for each element e of the output vector . The pooling operation is given as (3)

The output vector P_e is further compressed to obtain a compact vector c, by reducing the dimensionality via a fully connected (fc) layer, ensuring optimal efficiency in feature selection. (4)

In the above equation, φ represents the ReLU activation, ω represents Batch Normalization, and , where d is the depth of the vector.

Similarly, the Select operation selects different spatial scale information by the application of soft attention across the channels, guided by the compact vector c. Softmax is applied channel-wise such that for a two-branched network U₁, the mathematical operation to extract the attention score is given by (5)

The final feature map F_e is given by: (6) where, (7)

The above Eq (7) can be extended for a three-branched network U₂, used in this study, by calculating c_e through the third matrix C, acquiring the final feature map of U₂ as (8)

3.3 Dataset description

Existing 3D pose estimation datasets face a myriad of challenges, including limited diversity, ambiguous depth estimates, training data limitations, and predominantly indoor and controlled setups. These limitations impede progress in advancing 3D pose estimation algorithms to tackle the complexities of real-world scenarios. To somewhat bridge the gap in environmental diversity, this study utilizes two datasets, namely MPII and HumanEva-I, to familiarize the 3D pose estimation model with outdoor conditions. The MPII 2D Human Pose dataset [35] is widely used in computer vision due to its diverse and “in-the-wild” nature. This dataset contains around 25,000 photos capturing a wide range of human poses in different real-world scenarios. Each image is meticulously annotated with 16 different body joints, covering important anatomical regions.

The HumanEva-I dataset [36, 37] was created to evaluate and assess the performance of 3D pose estimation algorithms. It focuses on scenarios involving dynamic movements of multiple subjects and is annotated with 3D ground truth keypoints for each frame. It includes 15 anatomically significant joints such as the pelvis, hips, ankles, and thorax. The dataset includes seven video sequences—four in grayscale and three in color—featuring four annotated individuals engaged in five distinct actions. The MPII dataset is used for the initial training of the 2D pose estimation module, while HumanEva-I is employed for weakly supervised training of the 3D pose estimation pipeline. The MPII dataset’s skeleton topology contains 16 annotated joints, whereas HumanEva-I’s skeleton topology includes 15. Therefore, the neck joint is omitted during training for consistency.

3.4 Supervised pre-training on MPII

In this study, the first training phase of the entire 3D pose estimation pipeline involved using the MPII dataset to train the baseline 2D pose estimation model in a supervised setting. Pre-training on the MPII dataset aims to achieve optimal performance in realizing 2D poses in diverse outdoor environments, as these trained weights are later leveraged during the training of the 2D model on the HumanEva-I dataset. By increasing the generalization ability of the baseline 2D pose estimation model through pre-training on MPII, the second training phase on the HumanEva-I dataset, using parametric transfer, yields improved results, considering the indoor nature of the dataset and its MOCAP-based acquisition. High pose estimation accuracy from the 2D network is crucial for efficient self-supervised training of the 3D pose estimation model. The pose estimates generated by the 2D model are triangulated to obtain the respective 3D point estimates and used as cached ground truths for training the 3D pose estimation model.

3.5 Self-supervised training

Weak or Self-Supervised training refers to a training regimen where the data is not explicitly pre-annotated through manual labor; instead, alternative methods are employed for automated ground truth generation. All of them normally rely on some additional algorithms or properties of training data sets or their structures. While in simple and computational inexpensive tasks like age estimation, pose estimation it is pretty reasonable, but in sophisticated and time and computation consuming tasks such as 3D pose estimation, one depends on 3D annotated data and it forms bottleneck. Self-supervised methods give an efficient way to minimize the reliance on human inferences, which in the long run eliminates costs and manual efforts. In the current paper, we show a way to obtain the ground truth 3D poses for training the 3D pose estimation model in an unsupervised manner by utilizing trifocal tensors, which is explained in the subsequent section.

There are three views considered in giving unsupervised 3D ground truths required in training the 3D pose estimation module. The selection of three views is due to trifocal tensors which are used in the current study as a contrast to the fundamental matrix approach where two-view geometry is used. The trifocal tensors provide control of correspondence measurements as three-view geometry proposed by the trio-view approach, and, therefore, make the estimates of 3D pose more accurate. Also, problems of uncertainty in pose estimates of the objects are reduced inasmuch as stringencies inherent in the restraints make the estimate less vulnerable to degenerate solutions. To train the upper part of the pipeline, a trio of images L₁, L₂, and L₃ with a resolution of 728 × 728 from the HumanEva-I dataset are fed into both the frozen 2D pose estimation and the 3D pose estimation modules. The output pose estimates A_{1(x, y)}, A_{2(x, y)}, and A_{3(x, y)} from each trio are geometrically transformed using the trifocal tensor-based estimate to generate the 3D vector within the confines of the global coordinate frame. Similarly, the 3D pose estimation model outputs a vector containing the estimations for the joints in 3D space.

Under the pinhole camera projection model, let the p^th joint in the i^th image represent the 2D coordinates given by R_{(i, p)} = [x_{(i, p)}, y_{(i, p)}], while the respective 3D coordinates are given as Q_p = [X_p, Y_p, Z_p]. The relation of 3D points in homogeneous coordinates within the image plane for the pinhole projection model can be given by: (9) Here, (x_{(i, p)}, y_{(i, p)}, ω_{(i, p)}) represent the homogeneous coordinates on the image plane while (X_p, Y_p, Z_p) are the 3D points in the camera coordinate system. T and R are the extrinsic parameters that give the translation and rotation of the camera while K holds the intrinsic parameters of the camera inclusive of focal lengths f_x, f_y and the principal points c_x, c_y and is mathematically expressed as (10)

From the above definitions, the fundamental matrices F₁₂, F₂₃, F₃₁ for each pair of views, i.e., [1, 2], [2, 3], [3, 1] in a trio-view geometry, can be calculated as under (11)

The epipolar restraints of 2D coordinates for the trio-view can be defined as (12) (13) (14)

Using the trifocal tensor T_f to express the fundamental matrices, we get (15)

Given the points (R_1,p, R_2,p), the third epipolar line l_3,p can be computed using (16)

The 3D position of the joint R_3,p can now be triangulated by computing the point of intersection with respect to the joint in each view for all three epipolar lines. The output 3D coordinates for each point are used to supervise the training of the 3D Pose estimation model. During self-supervised training, the 3D pose estimation model predicts the 3D coordinates from the input image for each of the corresponding joints given by J_{(x, y, z)}, the loss is then calculated between the cached ground truths estimated using epipolar geometry and the predicted 3D key-points J_{(x, y, z)}. By projecting the predicted 3D key-points J_{(x, y, z)} into the camera space, the loss L_j can be computed by minimization of by substitution of . (17)

4.1 Evaluation metrics

In this subsection, we have given a brief introduction of all evaluation metrics we have utilized in this research work.

4.2 Experimental results and analysis

This paper introduces a novel approach to self-supervised training for 3D pose estimation by integrating trifocal tensor-based multi-view geometry with selective kernel networks for dynamic feature extraction. The pipeline works as follows: the left column of Fig 4 represents different video frames, the middle column approximates 2D key-points, and the right column displays the corresponding 3D poses obtained via triangulation. The proposed method demonstrates its efficacy with a Mean Per Joint Position Error (MPJPE) of 29.9 in a fully supervised setting and 47.6 in a weakly-supervised setting on the HumanEva-I dataset.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. Result comparison of proposed methodology with previous works on MPJPE metric.

https://doi.org/10.1371/journal.pone.0310831.g004

As shown in Table 2 and Fig 4, the benchmarks set by Pavlakos et al. [41] achieved an impressive MPJPE of 18.3 on the same dataset, with other methods like those proposed by Moreno-Noguer [42] and Martinez et al. [43] also delivering competitive results with lower MPJPEs. While our proposed pipeline shows better performance in a fully supervised scenario than earlier approaches, it still exhibits a slightly higher MPJPE compared to the work by Pavlakos et al. [41, 44]. The superior performance of [41] can be attributed to their use of depth maps, which provide additional guidance for generating 3D correspondences from monocular images, thereby reducing inaccuracies. In contrast, with an MPJPE of 47.6 in weakly-supervised training, our model demonstrates a significant capability to generate accurate 3D pose approximations from monocular images within a self-supervised pipeline. Additionally, the ground truths generated using trifocal tensors in a self-supervised setting offer a more robust reference for training the 3D pose estimation model, contributing to its overall effectiveness.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Result comparison of proposed methodology with previous works on HumanEva-I dataset.

https://doi.org/10.1371/journal.pone.0310831.t002

The fully supervised Tri-POSE-Net model achieves a lower MPJPE of 29.9 due to its access to extensive labeled data, enabling precise 3D pose estimation. It took around 49 seconds to gets trained and give results. In contrast, the weakly supervised model, with an MPJPE of 47.6, relies on trifocal tensor-based multi-view geometry to approximate 3D poses from limited data, sacrificing some accuracy for scalability. It took around 35 seconds to gets trained and give results. While the fully supervised approach excels in precision, the weakly supervised model offers greater adaptability in data-scarce environments, highlighting a trade-off between accuracy and data availability.

4.3 Experimental results and analysis

There are also noticeable performance discrepancies between the 3D posture estimation techniques in Table 3, with some exhibiting fewer errors and others showing a higher number of mistakes. Three methods, namely [42, 46, 47], stand out as particularly effective. They demonstrate consistently reduced Mean Per Joint Position Errors (MPJPE) across a range of movements and participants. Specifically, [41] performs exceptionally well, achieving competitive accuracy in both walking and running movements. Conversely, high error rates are exhibited by Bo et al. [48] and Simo-Serra et al. [49], indicating subpar performance. While the technique in [49] falters, notably in walking scenarios, [48] shows particularly high error rates in both walking and running. These results demonstrate the wide range of effectiveness in 3D pose estimation techniques, highlighting the need for thorough assessment and selection according to application-specific needs. Among the methods tested, our suggested approach—whether fully supervised or not—demonstrates encouraging outcomes. While the weakly supervised version achieves competitive performance across almost all action modes, the fully supervised version exhibits substantially lower errors. This implies that the above proposed approach could be useful in other scenarios and the given approach is a promising direction for further research in context to weakly-supervised 3D pose estimation under certain modifications and large-scale testing. However, the indicated research methodology opens significant potential for further improvement and fine-tuning, which can be traced from high efficiency estimates in the presence of different movement patterns. Though the knowledge gained from fully supervised and weakly supervised variants our proposed framework has the potential to perform well in different motion scenario. The obtained results are strengthen the concept of proposed approach for real conditions focusing to its feasibility for different motion scenarios. Moreover, the level of supervision that our method is capable of in its application stands out because of the variety in real-world data availability restrictions that can be addressed.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Detailed result comparison of proposed approach with other works on HumanEva-I dataset.

https://doi.org/10.1371/journal.pone.0310831.t003

4.4 Evaluation at different model depths and architectures

We performed a thorough analysis by comparing the performance of residual networks (ResNets) with varying depths and some other state-of-the-art models during the pre-training phase in order to better understand the effects of network depth fluctuation and architectural definition on the overall 3d-pose estimation pipeline. Our goal was to clarify how, in the context of 2D posture estimation, the depth of the network design affects the caliber of pre-training results. We discover that there is a significant relationship between PCK measures for 2D pose estimate attained in the pre-training phase and the effectiveness of model migration later on. In particular, we found that networks with higher accuracy during pre-training performed better after migration. This occurrence is especially important because it suggests that networks that perform better at the beginning of the transfer process are also proficient at using the learnt representations. The evaluation alluded to the synergy of ResNet-50 and Selective Kernel Network (SKNET) yielding superior performance in the first phase of training pipeline, i.e., the 2D keypoint estimate training on MPII dataset. Based on the quantitative outcomes, it was chosen as the backbone architecture instead of other backbone architectures like ResNet-18, ResNet-101, VGG-16, and MobileNet. Quantitative metrics such as Mean Squared Error (MSE) and Percentage of Correct Keypoints (PCK) employed within this study, illustrate the greater accuracy and robustness of ResNet-50 in comparison with other architectures. A comparative analysis of backbone architectures given in Table 4, indicates that deeper architectures such as ResNet-101 may result in increased computational overhead without achieving appreciable performance gains, while ResNet-18 may not have the depth and capacity to capture intricate spatial relationships required for accurate keypoint localization. On the other hand, ResNet-50 provides a convincing trade-off, by striking a compromise between model complexity and computational efficiency. In particular, ResNet-50 regularly performs better than alternative backbone architectures on benchmark datasets like COCO or MPII, obtaining lower MSE values and higher PCK scores at different thresholds [51]. Herein, ResNet-50 achieved an MSE of 15 and a PCK of 0.85 at a threshold of 0.5, surpassing the performance of ResNet-18 for the first training phase. Similarly, compared to ResNet-101, ResNet-50 exhibits comparable performance with lower computational cost, making it more suitable for real-time applications and large-scale datasets. Additionally, based on thorough quantitative evaluations and considerations of model complexity, computational efficiency, and feature representation, ResNet-50 emerges as the optimal candidate for integration into SKNet blocks for 2D keypoint estimation model training.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 4. Evaluation results for different backbone architectures.

https://doi.org/10.1371/journal.pone.0310831.t004

4.5 Ablation study

The ablation study (Table 5) investigates how the three components-sKL-1, SKL-2, and Weights Sharing—affect the accuracy of 3D pose estimation, as determined by Mean Per Joint Position Error (MPJPE). With these components removed, the baseline model has a comparatively high MPJPE of 78.7. It appears that this error can be brought down to merely 65. 9 when SKL-1 is introduced alone to show the effect that is separately brought about. Last but not the least, the level of accuracy is enhanced to 61%. We have got the added advantage of getting 1 by combining SKL-1 and SKL-2 that shows that there may be an interaction. The lowest MPJPE, 47. 6, is got when all componentspointing out the synergistic effect of the SKL-1, SKL-2 and Weights Sharing in enhancing the overall performance of the model is added. Thus, this ablation study is crucial in coming up with better-performing 3D pose estimation models.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 5. Ablation study results.

https://doi.org/10.1371/journal.pone.0310831.t005

4.6 Limitation of the proposed model

The following advancements have been done by the proposed model in the field of 3D human pose estimation. Nevertheless, it has been associated with a number of drawbacks. Firstly, it is noted that both the selective kernel networks and the self-supervision with trifocal tensors enhance the performance of the model, at the same time, adding the burden of complexity regarding computational needs and augmented training time. Furthermore, the accuracy and efficiency of the model depends on the heuristic algorithm, the kind of training data and the selected hyperparameters of the algorithm and these could restrict the model to the selected training data and might not work well on other sets of data and in the real world. Moreover, the proposed model is not robust for estimate the poses when the certain level of occlusions are present, the case of ambiguous configuration of the joints, and high variations in the appearance of the poses, these are the classic issues in tracking context. Last but not the least; the implementation as well as the deployment of the proposed model might need significant knowledge in computer vision and deep learning that limits the model’s applicability to researchers or practitioners who are not well versed in the field. The latter shows direction for the future work in order to improve the proposed approach’s stability, performance and user-friendliness in realistic applications.