2.1. RNN and CNN based action recognition methods

With the development of deep learning, RNN- and CNN-based action recognition methods have become the dominant way to solve the problem of skeleton-based human action recognition. RNN-based methods are able to capture long-term dependencies in the data and usually use coordinate vectors to model the skeleton data. Yong et al. [2] developed an end-to-end hierarchical RNN architecture, which has the advantage of processing time series data, enabling it to extract richer spatio-temporal features. Song et al [13] proposed an end-to-end spatio-temporal attention RNN that selectively focuses on the key spatial and temporal features to improve the understanding and recognition of complex actions. Zhang et al. [14] proposed a viewpoint-adaptive RNN with a Long Short-Term Memory (LSTM) network architecture that dynamically self-adjusts its orientation to skeletal data to the most optimal configuration in one end-to-end mode to enhance the performance of human action recognition.

In comparison, CNN-based methods are able to excellently deal with image and spatial aspects of data, and generally pseudo-images model the skeleton data. Liu et al. [15] proposed a multi-stream CNN fusion network based on a viewpoint-invariant approach for enhanced skeleton visualization, and by exploring the complementary properties between different types of enhanced color images, as a way to improve the performance of action recognition. Ke et al. [16] proposed a new method for 3D action recognition of skeletal sequences based on cylindrical surface coordinates. This method utilizes a multitask learning network to process feature vectors from all time frames in parallel for action recognition. Cao et al. [17] designed CNNs combining residual blocks and gated connectivity, which effectively utilized the ability of CNNs to process low-dimensional skeletal data and significantly improved the recognition accuracy of the network through the incorporation and improvement of residual blocks. Mumtaz et al. [18] studied a new dynamic frame skipping technique to generate meaningful temporal representations for spatio-temporal motion sequences, aiming to better detect anomalous actions. However, these RNN- and CNN-based methods cannot prove the graph structure of skeleton data well, resulting in certain missing skeleton data information when extracting features. Therefore, there is a need to find more effective methods for representing skeleton data.

2.2. GCN-based action recognition methods

GCN is a novel action recognition method in the field of deep learning, which utilizes graph structures to model the spatial relationships and temporal dynamics between human skeletons or joint points to efficiently capture the features of human actions. The method based on spatio-temporal graph convolutional networks (ST-GCN) was developed by Yan et al. [8]. This is the first attempt to employ GCN for modeling skeletal data. It enables the network to address spatial structure information as well as temporal dynamic information by extending graph convolution to the spatio-temporal domain, and also resolves the problem of traditional skeleton modeling that depends on handcrafted components or traversing rules, thereby enhancing the generalization and expressive power of the network. Shi et al. [9] have put forward a new type of two-stream adaptive graph convolutional network (2s-AGCN). It defines a dual-stream network with respect to spatial and temporal streams employing the adaptive graph convolution concept. This enhances the degree of freedom of graph models constructed and their applicability to a wide range of datasets. Ye et al. [19] proposed a novel Dynamic GCN (Dynamic GCN), an architecture that incorporates a Context Encoding Network (CeN) designed to learn context-rich dynamic skeleton topologies. CeN enables the modeling of dynamic graph topologies by integrating contextual information from other joints into the relationship between any two joints. Accordingly, the model is able to identify slight differences in the actions, resulting in improved performance in action recognition and enhancing the generalization ability of the model. Plizzari et al. [20] proposed a new Spatio-Temporal Transformer Network (ST-TR), which uses the Transformer’s Self-Attention Operator to capture dependencies between articulation points. The model effectively extracts the spatio-temporal features of the skeleton data in space and time through the Transformer architecture, which thus optimizes the performance related to the skeletal action recognition task. Song et al. [21] put forth a collection of efficient GCN baseline models (Efficient GCN) that incorporate increased accuracy while having a small number of trainable parameters, create stronger models through a spatio-temporal separation learning strategy, achieve faster baselines, and, further, lower the modeling complexity and over-parameterization. Nevertheless, despite the superiority of such models in performing on simple data relationships derived from pairwise interaction as well as graph neural networks, they exhibit limitations when dealing with higher-order data relationships and more complex structures.

2.3. Hypergraph neural networks

Hypergraph neural network (HGNN) [22] is a state-of-the-art data representation learning method designed to capture higher-order dependencies among data points. Unlike traditional graph neural networks, HGNN connects an arbitrary number of nodes via hyper edges, thus enabling the modeling of more complex data relationships than pairwise connections. Jiang et al. [23] proposed the Dynamic Hypergraph Neural Network (DHNN), which is capable of updating the hypergraph structure at each layer of the network to adapt to the dynamics of the action. Moreover, Gao et al. [24] developed a dynamic hypergraph representation and learning framework which effectively portrays the higher-order dependencies between nodes in the hypergraph structure through tensor representations, providing an important extension to existing hypergraph learning techniques. Mei et al. [25] proposed a Dynamic Hypergraph Hyperbolic Neural Network (DHHNN) based on variational autoencoder, which integrates hyperbolic geometry, dynamic hypergraph structure, and self-attention mechanism to dynamically adjust the contribution degrees of nodes and hyperedges, effectively enhancing the modeling capability for complex multimodal data. Overall, the hypergraph neural network through its unique structure, is able to effectively extract and represent higher-order features in skeleton data, thus demonstrating excellent performance in skeleton-based action recognition tasks.

3.1. Graph convolutional networks

In traditional GCNs, a skeleton sequence consists of single-frame skeleton graphs. The data is represented by a graph structure , where represents the set of joints and represents the set of edges. These edges include the skeletal edges , which connect corresponding nodes in each frame, and the edges , which link skeletal points across all frames. The feature matrix of the skeleton graph is represented by the adjacency matrix , whose element values reflect the connectivity between joints. The update rule of the graph convolutional network is defined as follows:

(1)

where is the input signal of the convolutional layer , and denotes the initial feature of the input layer, is the learnable weight matrix of the convolutional layer , refers to the nonlinear activation function, is the normalized graph adjacency matrix, and is the degree matrix of , while is the number of nodes in the graph, and is the node vector dimension.

3.2. Hypergraph convolutional networks

A hypergraph is defined by a triple consisting of a node set , a hyperedge set , and a hyperedge weight set , whose hyperedges are capable of connecting an arbitrary number of nodes for a more thorough representation of relationships. For the hypergraph , the association matrix is shown in Fig 2 and its elemental relations are defined as:

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. (a) Representation of a hypergraph convolutional network. Grey circles indicate different nodes and closed lines indicate different hyperedges. (b) Representation of the feature matrix of the hypergraph convolutional network.

https://doi.org/10.1371/journal.pone.0332066.g002

(2)

The degree of a given node denotes the number of hyperedges containing that node, denoted as:

(3)

where is the weight of the hyperedge and the degree of the hyperedge denotes the number of joints that constitute the hyperedge , denoted as:

(4)

The update rule for hypergraph convolutional networks is defined as [24]:

(5)

where is the weight matrix of the convolutional layer used to extract features for the nodes in the hypergraph. denotes the diagonal matrix of the node degree , indicates the diagonal matrix of the hyperedge degree , and is the diagonal matrix of the weights of the hyperedge.

3.3. Dual-branch differential channel hypergraph convolutional network

As presented in Fig 3(a), our proposed DBC-HCN model has two parallel processing streams, namely the ST-HCN stream and the CD-HCN stream. Spatio-Temporal Dynamic Hypergraph Convolution consists of spatial convolution module composed of a Dynamic Channel Refinement (DCR) module, a Hypergraph Feature Interaction (HFI) module, and a Feature Aggregation (FA) module, and a Dilated Temporal Convolution (DTC) module. Specifically, DCR module adjusts the weights of the feature channels through an adaptive mechanism to extract key features that are closely related to action recognition. HFI module extracts the representational differences with the help of the topologies of multiple hypergraphs, and FA is the fusion of information from both sides. DTC module has the purpose of widening the sensory field of the model such that the long-term dependencies that exist in the time-series data can be captured. Compared with the ST-HCN stream, a distinctive feature of CD-HCN stream is that the inter-channel difference operation is performed on the input feature maps, and an advanced representation of the differential features is learnt by feeding the obtained differential features into the convolutional network, which ultimately improves the discriminative power of the model.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. (a) The overall framework of DBC-HCN, the structure of CD-HCN and ST-HCN is similar, and the core difference is that CD-HCN uses a Channel Differential Mechanism(CDM) to extract differential features from the 3D channels of the input data. (b) Spatial Convolution Structure of DBC-HCN. (c) Dilated Temporal Convolution Structure of DBC-HCN.

https://doi.org/10.1371/journal.pone.0332066.g003

3.3.1. Spatio-temporal dynamic hypergraph convolution.

In this section, we explore the structural details of the spatial and Dilated Temporal Convolution (DTC) in the ST-HCN, and analyze the mathematical and architectural levels of HFI module, DCR module, and FA module, which make up the spatial convolution of the model.

Hypergraph Feature Interaction (HFI). Feature interactions between hypergraphs aim to reveal the deep connections between nodes in different hypergraph structures in order to understand and utilize the information of multiple hypergraph structures. To this end, we propose a novel feature interaction mechanism as shown in Fig 3(b). First, we apply linear transformation functions and to the input features, followed by characterization extraction via the Einstein summation convention function , as shown in Eq:

(6)(7)

where is the adjacency matrix of the static hypergraph, and denote the hypergraph adjacency matrices constructed by the K-NN and K-means algorithms [11,12,26], and represents the composite hypergraph adjacency matrix formed by the splicing operation of the adjacency matrices and . and are the output features after the interaction of inputs and hypergraph adjacency matrices and features.

Next, an element-by-element subtraction of the feature vectors of each corresponding node in different hypergraphs is performed by the function to obtain a difference feature matrix, Eq:

(8)

where is the nonlinear activation function and denotes the characteristic interaction matrix of the hypergraph.

After that, we first transform the features utilizing the linear function , then feed the transformed features into the function to complete the feature interaction with the interaction matrix , and finally use the interaction features to fuse the features as a way to obtain a more comprehensive interaction output, which is given by Eq:

(9)

where is a trainable scalar parameter and denotes the final output feature that contains the interaction information of all the above input features.

Dynamic Channel Refinement (DCR). We segment the Dynamic Channel Refinement process into three stages. First, the dynamic modeling and channel refinement are accomplished by inferring the hypergraph structure. Then, the transformation function is used for feature transformation. Finally, the channel topology is aggregated. In this process, denotes the channel correlation matrix and denotes the channel aggregation matrix.

(i) Dynamic modeling: Depicted in Fig 3(b), we begin by mapping the input features utilizing the linear transformation functions and , after which these dimensionality-decreasing features are fed into the dynamic modeling function . The dynamic modeling function is denoted as:

(10)

where is a nonlinear activation function, and centers on computing the distance between the node features and in the hypergraph along the channel dimensions and generating channel-specific representations of the topological relationships.

Next, we use a linear transformation on top of the modeling function to lift the features and learn the inter-channel correlation , Eq:

(11)

where is the adjacency matrix of the static hypergraph and the function is a multidimensional tensor algorithm.

Finally, to further refine the correlation between channels, we refine with channel correlation , computed as:

(12)

where is denoted as the channel topology after the refinement process, and is the Einstein summation convention.

(ii) Feature Transforms: we use feature transforms aiming to transform the input into a high-level feature representation via as shown in Fig 3(b). The graph convolution with a simple linear variation is used for this purpose with the following formula:

(13)

where is the transformed feature, denotes the input feature matrix, and denotes the shared weight matrix, which is responsible for linearly combining the input features during graph convolution to extract a feature representation rich in structural information.

(iii) Channel aggregation: under the premise of having channel topology and high-level features , the final output features can be obtained by aggregating the channel graph through the aggregation function with the formula:

(14)

where and come from the -th channel of with , the channel aggregation function according to the given channel topology and high-level features obtains the output matrix by channel aggregation of all connected channel graphs.

Feature Aggregation (FA). In this study, each of the four output features involved characterizes differentiated action characteristics among nodes in the network, and we aggregate the features as a way to comprehensively extract the relevance of these nodes and thus construct a more comprehensive global feature representation. The corresponding formula is:

(15)

where denotes the splicing operation.

Finally, on the basis of spatial convolution, we introduce a temporal convolution method for feature extraction for time series data, as shown in Fig 3(c). This study adopts the Dilated Temporal Convolution (DTC) strategy, which effectively improves the ability of feature extraction for time series data by flexibly adjusting the dilation coefficient.

3.3.2. Channel-differential hypergraph convolution.

In order to recognise and understand human movements more accurately, we propose a Channel-Differential Hypergraph Convolution method. The method is structurally similar to ST-HC, but adopts a novel Channel Differential Mechanism (CDM). The core idea of this mechanism is to extract the dynamic changes between channels, and thus to characterise the relative motion of skeletal points in a finer way. Compared with the traditional HCN and the ST-HCN proposed above, this method exhibits significant differences in feature extraction and information modelling, enabling the model to capture the dynamic changes between channels during the action more effectively.

In most hypergraph-based action recognition methods, HCN mainly relies on absolute coordinate information of skeletal points, such as X,Y,Z 3D coordinates. However, using only absolute coordinates may not be able to adequately portray the dynamic features of human motion.ST-HCN models the higher-order relationships of skeletal points by constructing spatio-temporal hypergraphs, but it still fails to explore the channel dimension in depth. As shown in Fig 4, Channel-Differential Hypergraph Convolutional Network (CD-HCN) is able to capture the subtle changes between neighbouring time frames by introducing a Channel Differential Mechanism, which enhances the representation of the dynamic features of the movement and thus improves the accuracy of recognition.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. Channel differential mechanism in channel-differential hypergraph convolutional network.

https://doi.org/10.1371/journal.pone.0332066.g004

In our approach, the core of the channel differential hypergraph convolution lies in first performing the difference operation on each channel of the input data, so as to realize the refined extraction of the features of the skeletal points, and then inputting the features into the model. Specifically, each skeletal point has a corresponding channel definition in the CD-HCN, and these channels represent the 3D coordinate information of the skeletal point respectively. By performing the difference operation on these channels, we obtain multiple single channels containing the 3D motion information of the skeletal points. This processing procedure is formally described by the following formula:

(16)

where is the number of input feature channels, and are the feature vectors of the -th and -th channels, and is the difference vector of the -th channel.

Compared with the traditional HCN that deals with the absolute coordinates of skeletal points directly, CD-HCN extracts neighbouring time frame change information via the Channel Differential Mechanism, enabling the model to focus on relative displacements during movement. This approach can eliminate the static differences between different individuals and improve the robustness of the model to complex movement patterns.

3.4. Model architecture

The architecture of the model proposed in this paper is shown in Fig 5, which consists of a two-stream network, where the second stream specifically integrates a Channel Differential Mechanism that focuses on analyzing frame-to-frame variations between channels. Both streams consist of ten layers, which include spatial convolution and temporal convolution. Spatial convolution aims to capture spatial information of joints and bones, while temporal convolution is used to extract temporal information in different frames. In spatial convolution, we incorporate Hypergraph Feature Interaction and Dynamic Channel Refinement to analyze static and dynamic hypergraph features. The convolution kernel size for temporal convolution is fixed to 5 × 1, except for the maximum pooling layer which is 3 × 1, and employs varying dilation rates to expand the receptive field. At layers 5 and 8, the temporal dimension is halved through stride temporal convolution. Subsequently, the model predicts action labels by global average pooling and FC layers.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. The overall architecture of DST-HCN.

The temporal modeling structures of ST-GCN and CD-GCN are similar.

https://doi.org/10.1371/journal.pone.0332066.g005

Finally, in order to fuse the information of the two streams for interaction, we use a late fusion strategy. Specifically, the features of ST-HCN stream are integrated with CD-HCN stream, Eq:

(17)

where , denotes human action data, and and denote spatio-temporal hypergraphic convolutional network and channel differential convolution, respectively.

4.1. Datasets

NTU RGB + D 60: NTU RGB + D 60 [27] dataset is a large-scale dataset widely used in the field of action recognition, which contains 56,880 skeleton action sequences performed by 40 volunteers covering 60 different action classes. Each action sample ensures that a maximum of two subjects are involved and is captured simultaneously by cameras from three different viewpoints, thus providing rich information about the 3D skeletal joint points. The authors of this dataset recommend two evaluation benchmarks: (1) Cross-subject (X-Sub) benchmark: the dataset is divided into two groups, with the training data coming from 20 subjects and the test data coming from the remaining 20 subjects, for a total of 40,320 training samples and 26,560 test samples. (2) Cross-View (X-View) benchmark: training samples come from camera views 2 and 3, totaling 37,920, while test samples come from camera view 1, totaling 18,960.

NTU RGB + D 120: NTU RGB + D 120 [28] dataset is a significant extension of the NTU RGB + D 60 dataset with the addition of 57,367 new skeleton sequences and 60 new action categories, making it the largest 3D co-annotated human action recognition dataset available. The dataset consists of more than 114,000 skeleton action sequences executed by 40 volunteers in 32 different settings, each representing a different location and context, to enhance the model’s ability to generalize across different environments. In order to evaluate the performance of the model, NTU-120 proposes two benchmark evaluation methods:(1) Interdisciplinary (X-Sub) Benchmark: same as the X-Sub benchmark of NTU-60, the X-Sub evaluation divides the dataset into two groups, one for training and the other for testing. (2) Cross-Set (X-Set): X-Set evaluation over splits the training and testing samples based on the name of the camera setup ID.

Kinetics-Skeleton: Kinetics-Skeleton [29] dataset is a large-scale human behavior recognition benchmark based on YouTube videos, containing about 300,000 video clips covering up to 400 human behaviors involving daily activities, motion scenarios, and complex human-computer interactions. The original Kinetics dataset only provides raw video clips without skeleton sequences.ST-GCN [2] obtains the positions of 18 joints on each frame of the video by applying the publicly available Open-Pose toolkit, and accordingly selects the two individuals with the highest average joint confidence for skeleton data extraction. The processed skeleton data were divided into 240,000 training clips and 20,000 validation clips.

4.2. Implementation details

All experiments were conducted utilizing the PyTorch deep learning framework, in which we implemented stochastic gradient descent (SGD) with momentum of 0.9 as the optimizer, and selected cross-entropy as the loss function for the backpropagated gradient, with weight decay configured at 0.0001. During the training phase on NTU RGB + D 60 [27] and NTU RGB + D 120 [29], the learning rate was increased at the 35th and 55th epoch, with the training process reaching completion at the 80th epoch. For Kinetics-Skeleton [22], the learning rate demonstrated decreases at the 45th and 55th epochs, and the training process reached completion at the 70th epoch. For the NTU RGB + D 60 and NTU RGB + D 120 datasets, the batch size is set to 70. For the Kinetics-Skeleton dataset, we set the batch size to 64.

4.3. Ablation study

In this section, to validate the effectiveness of our proposed module and dual-stream framework, we conduct the following experiments using the X-Sub benchmark on the NTU RGB + D 60 dataset.

In multi stream fusion, we integrate different output features, specifically J, B, and M (including joints, bones, and motions). In order to evaluate the effect of each stream on the model performance, we implemented three sets of controlled experiments, as shown in Table 1. The results showed that the B (bones) stream had the most significant impact, with a performance gain of up to 2.6%, thereby further improving the recognition accuracy.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Comparison of performance with different input data.

https://doi.org/10.1371/journal.pone.0332066.t001

In order to validate the effectiveness of the DCR module, HFI module, and the dual-stream methods of ST-HCN and CD-HCN, we implemented a removal-by-removal ablation study to analyze the DBC-HCN model. By monitoring the performance changes in the B-stream, we evaluate the effectiveness of these modules, as shown in Table 2. In the DBC-HCN w/o HFI model with HFI removed, we observe a 0.6% performance degradation, thus confirming the critical role of HFI between channels. In addition, the DBC-HCN w/o DCR model shows a 2.0% performance degradation, which highlights the importance of DCR in improving model performance. Meanwhile, the performance of DBC-HCN w/o CD with removal of CD-HCN and DBC-HCN w/o ST model with removal of ST-HCN decreased by 1.4% and 1.2%, respectively, indicating the vital role of the dual-stream interaction in improving model performance.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Comparison of performance when adding or removing HFI, DCR, ST-HCN or CD-HCN from DBC-HCN.

https://doi.org/10.1371/journal.pone.0332066.t002

4.4. Comparative study

This section presents a comparative analysis of the DBC-HCN model’s performance against state-of-the-art skeleton-based action recognition methods. We will test it on different benchmarks on the NTU RGB + D 60, NTU RGB + D 120, and Kinetics-Skeleton datasets.

Comparative data for the models are shown in Tables 3, 4 and 5. Across all three datasets, our method proves superior to the vast majority of methods when evaluated against almost all metrics. Specifically, regarding the NTU RGB + D 120 dataset, our model, which combines joint and bone information, reaches state-of-the-art performance levels. The DBC-HCN model surpasses the current hypergraph model DST-HCN [37] on two evaluation benchmarks, indicating improvements of 0.6% and 0.5%, respectively.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Performance comparison of the state-of-the-art methods of DBC-HCN on the NTU RGB + D 60 dataset.

https://doi.org/10.1371/journal.pone.0332066.t003

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 4. Performance comparison of the state-of-the-art methods of DBC-HCN on the NTU RGB + D 120 dataset.

https://doi.org/10.1371/journal.pone.0332066.t004

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 5. Performance comparison of state-of-the-art methods for DBC-HCN on the Kinetics-Skeleton dataset.

https://doi.org/10.1371/journal.pone.0332066.t005

We used the X-Sub benchmark on the NTU RGB + D 60 dataset as a challenging testbed for evaluating model performance. Fig 6 illustrates the confusion matrix of the DBC-HCN model on the NTU RGB + D 60 X-Sub benchmark. Fig 6 shows the confusion matrix of the DBC-HCN model at NTU RGB + D 60. By analyzing the confusion matrix based on X-Sub, we can see that although some classes are significantly less accurate than others, our method can accurately identify most of them. Fig 7 further evaluates this trend. BCD-HCN has an accuracy rate of over 90% in 38 action categories, accounting for 63.33% of all action categories, indicating that the model can accurately identify actions in most cases. In addition, the model has an accuracy rate of over 80% for 52 action categories, accounting for 86.66% of all action categories. These statistical data highlight the outstanding accuracy and reliability of BCD-HCN in action recognition tasks.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 6. Confusion matrix for the NTU RGB + D 60 dataset.

The yellower the square on the diagonal, the more accurate the identification. (a) X-sub benchmark on the NTU RGB + D 60 dataset. (b) X-view benchmark on the NTU RGB + D 60 dataset.

https://doi.org/10.1371/journal.pone.0332066.g006

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 7. The accuracy of x-sub and x-view in NTU RGB + D 60 for this work.

https://doi.org/10.1371/journal.pone.0332066.g007

Among the 60 action categories, the model demonstrates particularly strong performance in recognizing actions such as “take off jacket”, “jump up”, “hopping”, “staggering” and “walking towards”. We can attribute the high accuracy for these actions to their unique visual features and the model’s successful capture of these characteristics. However, the Staple action recorded the lowest recognition accuracy.