Density map-based methods

For a given crowd image, density map-based methods aim to generate a density map and then sum the predicted density map to obtain the count [27–31,47,48]. Specifically, Li et al. [27] proposed the Congested Scene Recognition Network (CSRNet) model, which uses dilated kernels to provide a larger receptive field to generate high-quality density maps, thus achieving high-precision crowd counting. Liu et al. [47] introduced a nonlinear continuous counting quantization strategy and transformed the problem of sample block counting imbalance into a class imbalance of counting levels. Xiong et al. [28] proposed the Spatial Divide-and-Conquer Network (S-DCNet), whose core idea is to learn a counting classifier on a closed set and then extend it to open-set counting. Miaott et al. [29] proposed a Shallow Feature-based Dense Attention Network (SDANet), capturing multi-scale information through densely connected hierarchical feature maps, achieving crowd counting for static images. Bai et al. [30] proposed a dilation rate adaptive convolution operation, establishing a self-correcting supervision mechanism to improve the accuracy of crowd counting. Liu et al. [31] proposed a local counting map, constructing a scale-aware module, hybrid regression module, and adaptive soft region module to achieve high-precision counting regression by focusing on the difference between the global sum of the crowd count and the density map during testing. Liu et al. [48] transformed the counting problem into a sequential decision problem, using a weighing strategy from scales to achieve a crowd counting model based on deep reinforcement learning.

Most existing density map-based crowd counting methods mainly focus on crowd density estimation. Relatively few works focus on generating crowd density maps. Ma et al. [32] proposed a crowd counting model that extensively leverages representations learned during encoding to compute first-phase multiscale features, and its decoder further fuses these scale-aware features to generate the density map. Wan et al. [33] analyzed the impact of different density maps and constructed a density map refinement network. They built an adaptive density map generator, using annotation dot maps as input to learn the density map representation of the counter, generating ground-truth density maps. Xu et al. [49] extracted patch-level density maps through a density estimation model, introducing multi-polar center loss to automatically normalize each patch density map online, achieving density map clustering. Jiang et al. [50] constructed a multi-level convolutional neural network that adaptively learns multi-level density maps, each focusing on handling pedestrians of specific sizes and fusing them to predict the final output. Tian et al. [35] proposed a more intuitive and understandable Density Map Dynamic Refinement Network (DDRNet), consisting of a counter and refiner. The refiner, composed of convolution layers with different dilation rates, iteratively refines and improves the quality of the density map using the counter’s output as dynamic input. Liu et al. [36] constructed a dynamic fine density map network with a designed regional attention module (RAM) that adaptively adjusts the head size relationship in different positions of the dot map, refining existing ground-truth density maps through joint training of the counter and learnable refinement network.

These methods have advanced crowd counting techniques but have yet to meet practical requirements. Firstly, manually annotating crowd density maps is challenging. Additionally, these methods heavily rely on the quality of crowd density maps. In practice, the density maps generated by existing methods are easily affected by changes in head proportions due to the multi-scale nature of pedestrians in images, failing to reflect the actual size of heads in the image, thereby impacting counting accuracy.

Localization-based methods

Localization-based methods utilize object detection techniques [51–53] for crowd counting, with the core idea of treating pedestrians or heads as visual targets and counting the crowd by locating these targets. Gao et al. [17] proposed using the cascade Adaboost algorithm to replace the candidate region generation module in the R-CNN framework [54], using convolutional networks to learn head visual features and employing a support vector classifier for pedestrian and non-pedestrian classification. Khan et al. [18] proposed using a scale map to generate scale-adaptive head candidate regions, followed by using convolutional neural networks for head detection. Rani et al. [19] first used Faster-RCNN [55] to detect heads and then built a pedestrian counting system. Additionally, the YOLO series object detection frameworks [20] have been widely applied in pedestrian detection and crowd counting tasks [21–26]. Sam et al. [34] developed a tailored detection framework for dense crowd counting by predicting head bounding boxes, using a top-down feature modulation strategy to better distinguish pedestrian targets and producing fine-grained predictions at multiple resolutions, reliably outputting head localization results from sparse to dense crowds.

Moreover, some point-based methods have been applied to crowd counting. The primary task of point prediction is to use supervised deep learning methods to achieve pedestrian target point regression based on pedestrian (particularly head) point annotations, thereby providing individual localization positions. Song et al. [40] proposed a Point to Point Network (P2PNet), a classic architecture among such methods, directly predicting a set of point proposals to locate head positions. Yuan et al. [41] constructed a Localization Guided Transformer (LGT) framework, a point-based model using regression heads and classification heads to simultaneously predict head point proposals and point confidence, providing more discriminative representations for high-quality density map estimation. Ma et al. [42] proposed a self-attention guidance-based crowd localization and counting network (SA-CLCN), using original point annotations from crowd datasets as supervision to train the network, predicting each head’s center point coordinates and the crowd count. Liu et al. [43] proposed a bipartite matching-based point-supervised crowd counting method, matching annotated pixel points through bipartite matching to reduce the impact of incorrect point matching on counting performance. Wang et al. [44] proposed a novel self-training method, using point-level annotations and crowd-aware loss to guide network training, predicting pedestrian center points and sizes in crowded scenes. Ma et al. [45] proposed a density contribution probability model based on Bayesian loss and point annotations, where the training loss constrains not the value of each pixel in the density map but the expected count of each annotated point, improving crowd counting accuracy in dense scenes.

Contrastive learning

The concept of contrastive learning was proposed as early as 2006 to learn invariant representations of patterns [56]. Subsequently, Khosla et al. [57] developed supervised contrastive learning, whose core idea is to cluster the embeddings of positive samples while separating the embeddings of negative samples. Supervised contrastive learning has significantly enhanced the learning ability of image feature representations. Recently, contrastive learning methods have gradually been applied to pedestrian detection and crowd counting. For instance, Lin et al. [58] proposed an example-guided contrastive learning framework to guide feature learning. Under the contrastive learning framework, they used pedestrian appearance as a prior knowledge example dictionary, constructing effective contrastive training pairs and using the constructed example dictionary to evaluate the quality of candidate pedestrians. Chen et al. [59] proposed a discriminative feature learning framework for crowd counting, consisting of a Masked Feature Prediction Module (MPM) and a Contrastive Learning Module (CLM). MPM randomly masks feature vectors in the feature map, enhancing the model’s pedestrian localization ability in high-density areas through a supervised reconstruction strategy. CLM brings the representations of pedestrian targets closer and pushes the background features away from pedestrian representations, increasing their discriminability. However, these methods do not solve the problem of crowd counting in both sparse and dense crowd scenarios. Unlike previous works, our method combines patch-level foreground and background contrastive learning with a point detection framework, providing more accurate results in various applications.

Contrastive learning network design

The objective of this section is to construct positive and negative samples for contrastive learning. Given a crowd image, we randomly crop a fixed number of 128 × 128 image patches, which may or may not contain people. Our key insight lies in aggregating the feature representations of crowd regions while separating those of background regions. By batching the image samples, we can highly likely obtain a series of samples that simultaneously include both crowd and background regions. These samples can then be utilized as positive and negative samples to establish a contrastive learning framework. For instance, when the batch size is set to 16 and the number of patches is 8, each training iteration will generate 128 samples, either containing or excluding human elements, which are applicable for the contrastive learning process.

Similar to existing crowd counting models [40], we can use VGG-16_bn [60] as the backbone network for extracting image features. As shown in Fig 2, by outputting three different levels of feature maps, we design a Multi-Scale Feature Fusion Module (MSFM) to obtain better feature representations of crowds of varying scales and distributions. Subsequently, based on the same output of the MSFM, we construct three different head branches. During the training phase, we adopt a linear projection layer as the projection head, and utilize three stacked convolutional layers interwoven with ReLU activation as the classification head and regression head. During inference, only the classification and regression branches are maintained for point prediction.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Overall framework of point-based crowd counting with contrastive learning.

The blue dashed box includes four different-sized feature layers from the VGG backbone network. The backbone network section can be replaced with other structures such as ResNet. The dashed box containing the projection head, contrastive loss, and point matching is used only during the training process.

https://doi.org/10.1371/journal.pone.0327397.g002

Multi-scale feature fusion module

The designed Multi-Scale Feature Fusion Module (MSFM) is shown in Fig 3. It takes three feature maps as input and outputs a fused feature map. The input feature maps , , and have sizes H/4xW/4, H/8xW/8 and H/16xW/16 respectively. The output feature map has a size of H/8xW/8. We first utilize FPN [61] to introduce top-down and bottom-up paths, obtaining three new feature maps with stronger semantics and finer information, namely L, M, and H. Considering the limitations of FPN in multi-scale feature fusion, we further introduce two additional connections. The mathematical description of the new feature maps is as follows:

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Structure of the multi-scale feature fusion module.

L, M, and H represent Low-level, Medium-level, and High-level features, respectively. W1, W2, and W3 are learnable weights for features at different levels. The symbol Σ denotes the element-wise weighted summation operation.

https://doi.org/10.1371/journal.pone.0327397.g003

(1)

where Conv2d is the standard two-dimensional convolution operation, Upsample is the upsampling operation, and the Swish activation function [62] is chosen for better performance, specifically defined as:

(2)

Here, is a constant or trainable parameter, set a in our network module.

For cross-scale connections, we additionally add a bottom-up path from the L level to the M level, using a MaxPool operation with stride 2 to adjust the output size of the L level to the same shape as the M level. Moreover, we add an extra edge connection in to sum operations of the same level as the M output, enabling the fusion of more features without incurring significant costs. Due to different input features at different resolutions, their contributions to the output features are typically unequal [63]. To address this, we perform a weighted summation operation on the inputs , M, and L, with the final fusion formula as follows.

(3)

Point matching and prediction

Similar to P2PNet [40], we use a one-to-one matching strategy for point matching. A set of fixed reference points is introduced. These reference points are densely and uniformly arranged over the patch obtained from the input image crop, with coordinates represented as . The classification branch is designed to output the confidence scores of these point coordinates, while the regression branch generates the offsets for these point coordinates. For a predicted point , the offset is represented as , and the predicted point coordinates are calculated as follows.

(4)

where is a regularization parameter used to adjust the magnitude of the offset.

For each point proposal in the set , a one-to-one matching strategy is employed, assigning them to the ground truth target set P. Based on the Euclidean distance and confidence score of these points, a pairwise cost matrix D is constructed, defined as follows.

(5)

where is a weighting parameter, and is the confidence score of .

Using the above pairwise cost matrix, the Hungarian algorithm [64,65] is employed for matching. In our implementation, the total number of reference points exceeds the number of pixels in the image, ensuring that the number of predicted points exceeds the number of ground truth points. Thus, matched predicted points can be labeled as positive samples, while unmatched predicted points are labeled as negative samples.

Loss function design

In our framework, there are three loss objectives: classification loss, regression loss, and contrastive loss. The first two are based on the matched and unmatched points obtained above, while the last one is calculated based on the deep feature maps output by the MSFM.

For the classification loss, we adopt focal loss [66] for dense target detection, which is described as follows.

(6)

where is the model-estimated probability for the label being 1, is a weighting parameter to address class imbalance, is a focusing parameter that reduces the loss contribution from easy examples and extends the range of examples receiving low loss. MMM represents the mini-batch size.

For the regression loss, we introduce MSE loss to supervise the point regression, defined as follows.

(7)

where denotes the position of the ground truth point, denotes the point matched with the ground truth point , and N is the number of matched points.

The contrastive loss is calculated on the feature maps output by the MSFM. Since we use random cropping to obtain patches during training and input them into the network, the first dimension of the output feature map is the product of batch size and the number of patches, denoted as . Let be the index of any sample in , be the index set excluding itself, and be the index set of other aligned visual features with the same label as . The contrastive loss for a training process is defined as.

(8)

where is the temperature hyperparameter, and denotes dot product. is the cosine similarity between sample and sample , and are the high-dimensional feature vectors of samples and , respectively. Therefore, the overall loss function can be formulated as follows.

(9)

where the weights for these loss functions , , are hyperparameters. In our experiments, we set to 0.01, to 1, and to 2e-4.

Experimental details

We utilize multiple publicly available crowd counting datasets, such as UCF_CC_50 [67], ShanghaiTech Part A and Part B [68], to demonstrate the superiority of our method. The UCF_CC_50 dataset contains 50 images with the number of people ranging from 94 to 4543, with an average count of 1280. This dataset is diverse in scene distribution and challenging for detection. We use the 5-fold cross-validation method, similar to other papers. The ShanghaiTech Part A dataset is randomly collected from the internet, containing 300 training images and 182 test images. Part B is captured from busy streets in the Shanghai metropolis, with 400 training images and 316 test images. The crowd density varies significantly between Part A and Part B, making this dataset more challenging and representative than most existing datasets. Additionally, we collected a non-public Tower dataset for practical mid-to-high viewpoint video surveillance applications, consisting of 671 training images and 100 test images.

Similar to previous works, we apply random scaling, keeping the shorter side no less than 128. We then randomly crop 8 fixed-size 128 × 128 image patches from the resized images. Finally, we introduce random flipping with a probability of 0.5. The Adam algorithm with an initial learning rate of 1e-4 is first used for training, followed by switching to the SGD optimizer for fine-tuning the better model [69]. The backbone network is pretrained on ImageNet, similar to previous works.

For the contrastive cropping sample parameter settings, we focus on the training data batch size and the number of patches per image. We set the batch size to 8 and the patch number to 3. Therefore, for each training process, we obtain 24 samples per batch, with labels indicating whether they contain people or not, which can be used for the contrastive learning process.

Model evaluation

Our method has been compared with state-of-the-art methods on several publicly available crowd counting datasets, as shown in Table 1. We use Mean Absolute Error (MAE) and Root Mean Squared Error (MSE) as evaluation metrics, defined as follows.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. The overall performance of our framework.

https://doi.org/10.1371/journal.pone.0327397.t001

(10)

where is the number of test images, is the actual number of people in the -th image, and is the estimated number of people in the -th image. Roughly speaking, MAE measures the accuracy of the estimates, while MSE measures the robustness of the estimates.

Our framework achieves the best performance among all methods. Specifically, on the dense crowd dataset SHTech Part A, our method reduces the MAE by 1.4 and the MSE by 1.16 compared to the state-of-the-art method P2PNet [40]. On the sparse crowd dataset SHTech Part B, our method also achieves the best performance, reducing the MAE by 0.12 compared to the state-of-the-art method RSI-ResNet50 [70]. On the challenging UCF_CC_50 dataset with a wide range of crowd densities, our method balances accuracy and robustness. Compared to P2PNet, our method reduces the MSE by 7.3 while maintaining a similar MAE, and compared to CAN, our method reduces the MAE by 38.7 while maintaining a similar MSE. Considering both MAE and MSE metrics, our method not only achieves high accuracy but also demonstrates high robustness, achieving the best overall performance.

As shown in Fig 4, we present the visualized crowd counting results based on the ShanghaiTech Part A and Part B datasets. In the ShanghaiTech Part A dataset, we observe densely crowded regions, while the ShanghaiTech Part B dataset is more sparsely populated. The images in ShanghaiTech Part A have higher densities, with the number of people ranging from 33 to 3139, whereas the images in ShanghaiTech Part B have lower densities, with the number of people ranging from 9 to 578 [36]. Due to the varying object shapes and sizes in these datasets, our multi-scale feature fusion model demonstrates significant advantages over traditional methods. Experiments on these datasets demonstrate that our method performs well in both sparse and crowded crowd scenarios.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. Visualization of our method on the ShanghaiTech Part A and Part B datasets.

The left image is from ShanghaiTech Part A, with a predicted crowd count of 375; the right image is from Part B, with a predicted crowd count of 18. All faces are blurred in Fig 4 for privacy preservation.

https://doi.org/10.1371/journal.pone.0327397.g004

Additionally, we evaluated our model using the Tower dataset. As shown in Fig 5, our method accurately predicts the number of people in both near-field images with fewer people and far-field images with more people. This indicates that our algorithm is adaptable to both near-field and far-field, as well as sparse and dense crowd scenarios.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. Example visualization results of our method on the Tower dataset.

The left image shows a close-up view of a high-speed rail station exit, with a predicted crowd count of 2; the right image shows a distant view of a street, with a predicted crowd count of 33. All faces are blurred in Fig 5 for privacy preservation.

https://doi.org/10.1371/journal.pone.0327397.g005

Ablation experiments

Considering that the main innovations of the paper revolve around contrastive learning and the multi-scale feature fusion, our ablation experiments focus on four aspects: the effectiveness of the projection head and contrastive loss related to contrastive learning, the effectiveness of the multi-scale feature fusion module, and the number of patches cropped from a single image.

Effectiveness of the projection head.

Given that different types of projection heads have varying performance, we conducted an ablation study on the structure of the projection head, including three settings: identity mapping, linear projection head using a fully connected (FC) layer of size 256*128, and nonlinear projection head using a 256*256 FC layer, a 256*128 FC layer, and a ReLU activation in between. As shown in Table 2, under the same framework with the patch number set to 4 by default, the linear projection head shows a certain advantage. The learnable linear transformation between the representations and the contrastive loss significantly improves the quality of learned representations.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Ablation study on projection head.

https://doi.org/10.1371/journal.pone.0327397.t002

Effectiveness of the multi-scale feature fusion module.

To analyze the impact of feature fusion, we compared the original FPN module with our Multi-Scale Feature Fusion Module (MSFM). Using the same VGG backbone network and the default hyperparameter settings mentioned above, the experimental results are shown in Table 3. The multi-scale feature fusion module reduces the MAE by 1.18 and the MSE by 2.45 compared to the original FPN module. As we can see, by replacing FPN with our proposed MSFM, we achieved better crowd counting performance. The results indicate that multi-scale feature fusion can extract scale-related features from crowd images, facilitating crowd detection in different contexts.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Evaluation of the effectiveness of MSFM.

https://doi.org/10.1371/journal.pone.0327397.t003

Effectiveness of the contrastive loss.

We conducted an ablation study on the use of contrastive loss, and the analysis results are shown in Table 4. To ensure the reliability of the experiments, the baseline model was based on the same model without contrastive loss, with the patch number set to 4 by default. We can see that adding contrastive loss reduces the MAE by 0.52 and the MSE by 1.16, showing that the model with contrastive loss performs better.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 4. Evaluation of the effectiveness of contrastive loss.

https://doi.org/10.1371/journal.pone.0327397.t004

Patch number parameter analysis.

Under uniform parameter configurations, experiments were conducted on the SHTechA dataset to investigate the influence of the contrastive cropping module on crowd counting by varying the number of patches cropped from a single image. As depicted in Fig 6, when the patch number is set to 3, the crowd counting accuracy, measured by mean absolute error (MAE), attains an optimal value of 51.34. This outcome is attributable to the ability of a moderate patch count to enable the contrastive learning framework to effectively capture multi-scale contextual information while maintaining a balanced distribution of positive/negative sample pairs, which is crucial for learning discriminative features in crowd scenes. As the patch number exceeds 3, significant fluctuations in the MAE metric emerge, primarily due to the introduction of redundant or semantically similar regions from excessive patches, which dilutes the informativeness of contrastive pairs. In contrastive learning, the efficacy of the loss function hinges on meaningful semantic disparities between sample pairs; thus, an overly high patch number forces the model to learn from highly correlated patches, leading to unstable gradient updates and degraded feature representation quality. Specifically, when the patch number surpasses 8, the overload of intra-image patches disrupts the equilibrium of the contrastive learning objective, causing the MAE to exceed the baseline value of 52.36. These findings underscore the necessity of constraining the patch number within a range below 8 to optimize contrastive feature diversity while mitigating information redundancy, thereby validating the theoretical rationale of the proposed contrastive cropping module. By adaptively selecting an appropriate number of patches, the module balances multi-scale feature extraction with contrastive sample quality, enhancing the discriminative power of learned representations and consequently improving crowd counting performance.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 6. Evaluation of MAE accuracy metric based on different patch numbers for our method on the SHTechA crowd counting dataset.

Patch number parameter is the number of samples cropped from a single image for contrastive learning.

https://doi.org/10.1371/journal.pone.0327397.g006

Qualitative study on contrastive learning

To obtain more insights into the effectiveness of contrastive learning, we apply t-SNE [78] to MSFM features by different models to compare their capability to distinguish between positive samples and negative samples. Fig 7 presents t-SNE maps for the baseline model and our model based on contrastive learning. The t-SNE map of features after the MSFM module integrated with contrastive learning shows more separability than the baseline t-SNE map.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 7. The t-SNE map of MSFM features by different models, in which the blue dots and red dots refer to the positive samples and negative samples.

Left: features after the MSFM model integrated with contrastive learning; Right: features of the baseline model.

https://doi.org/10.1371/journal.pone.0327397.g007

Complexity analysis

Table 5 reports a comparison of model size and inference speed computed with a single NVIDIA T4 GPU. The inference time is the average time of 100 runs on testing 1920 × 1080 sample. Those test images are collected from real-world monitoring scenes. It can be observed that our method has less parameters and faster inference speed than the previous state-of-the-art method STEERER [77]. Evidently, our model excels in addressing real-world scenarios in terms of both resource consumption and inference performance.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 5. Comparison of the Parameters (M) and Inference speed (s/100 images).

https://doi.org/10.1371/journal.pone.0327397.t005