LiDAR-based 3D detection

LiDAR-based 3D object detection is crucial in autonomous driving. Various methods have been proposed to process LiDAR point cloud data for accurate object detection and localization. Early research focused on point-based methods, such as PointNet [36,40] and SparseConvNet [41], which directly process raw point clouds. While these methods handle irregular point cloud data well, they are relatively slow and computationally inefficient. To improve speed and efficiency, voxel-based methods emerged, discretizing point clouds into regular 3D grids for efficient sparse convolutions. Notable works, such as VoxelNet [42] and SECOND [43], significantly enhanced computational efficiency by employing sparse convolutions and generating prediction results in the BEV space.

In addition to point-based and voxel-based approaches, range view representation has also gained attention in LiDAR 3D object detection [44]. This representation projects point clouds into an image-like space, simplifying computations. However, it often loses geometric detail, leading to suboptimal performance compared to other methods. Recent research has also introduced anchor-free designs, which eliminate the need for traditional anchor boxes, enhancing object representation and flexibility. Furthermore, Transformer architectures have been applied to LiDAR-based 3D detection [45,46], particularly for feature extraction and object representation, showing promising results despite higher computational complexity.

Camera-based 3D detection

Due to the high cost of LiDAR sensors, camera-based 3D perception has received significant attention. Camera-based 3D object detection methods typically operate using single-view or multi-view images. In the case of single-view images, some approaches attempt to predict 3D bounding boxes directly from image features or leverage intermediate representations for object detection [47,48]. For multi-view inputs, image features are optimized within a constructed 3D geometry volume, or projected and merged into the frustum space with predicted depth. However, since the accuracy of predicted depth maps is inferior to LiDAR, this leads to semantic ambiguity in the BEV space. To address this issue, many methods use geometric cues, such as extracting geometric information from multi-view images in an implicit manner, though this approach sacrifices direct spatial interactions.

Recent studies have introduced view transformation methods, using view transformers to convert camera features from perspective to BEV space [49,50]. These methods, inspired by LiDAR-based designs, aim to enhance the accuracy of 3D detection by transforming from perspective to BEV. For instance, BEVDet [51] and M2BEV [52] extend the LSS and OFT frameworks, incorporating explicit supervision for depth estimation. However, the accuracy of depth estimation remains a challenge for purely camera-based detection. Some methods, such as pseudo-LiDAR techniques, have been proposed to mitigate this limitation by converting RGB images into point clouds using pre-trained depth estimation networks [56]. These point clouds can then be integrated with standard LiDAR detectors. Another approach involves converting camera features into BEV representations, such as BEVFormer [49] and Bevformerv2 [50], which use spatiotemporal cross-attention to compute feature interactions across frames, enabling more efficient 3D object detection.

In summary, while camera-based methods have made significant progress in 3D object detection, they still face challenges with depth estimation and semantic ambiguity. The integration of geometric cues and the use of view transformers show promise in improving performance, but challenges remain when compared to LiDAR-based methods. The fusion of camera and LiDAR data is often seen as an effective way to address these limitations.

Multi-modal fusion for 3D detection

With the development of multi-sensor fusion technology, multi-modal fusion for 3D object detection has become a research hotspot [57–59]. By combining data from different sensors, such as LiDAR and cameras, the strengths of each sensor can be leveraged to enhance 3D object detection performance. These methods can be classified into proposal-level fusion, point-level fusion, and intermediate-level fusion. Each of these fusion strategies presents distinct trade-offs. Proposal-level fusion supports object-centric reasoning and efficient feature extraction but often depends on accurate region proposals and suffers from limited spatial alignment precision. Point-level fusion provides detailed semantic augmentation by projecting image features onto point clouds; however, it is susceptible to projection noise and lacks global context integration. Intermediate-level fusion strikes a balance by performing feature interaction in the BEV or semantic space, enabling richer cross-modal understanding while preserving structural consistency. Moreover, recent research efforts have emphasized the importance of flexible and context-aware fusion mechanisms across multiple domains. For instance, Wang et al. [60] proposed a spatio-temporal deep learning model for streamflow prediction based on multi-source data fusion, highlighting the benefit of integrating heterogeneous data sources. Similarly, Wang et al. [61] developed a hybrid Autoformer-ELM framework to enhance multi-step carbon price forecasting using diverse influencing factors. These studies, although focused on environmental prediction tasks, reinforce the value of adaptive and dynamic fusion architectures — a principle that underlies our design of LGMMfusion’s deformable alignment and weighted fusion strategy.

Proposal-level fusion methods were first introduced in MV3D [62], which generates object proposals in 3D space and projects them onto images to extract Region of Interest features. Several improvements have since been proposed, such as Frustum-PointNet [31] and Frustum-ConvNet [63], which elevate image proposals to 3D frustums to identify relevant regions in point clouds. FUTR3D [64] and TransFusion [65] generate object queries in 3D space and fuse image features into these proposals. These methods are object-oriented and suitable for object-level tasks, but are difficult to apply directly to other tasks, such as BEV map segmentation. Point-level fusion methods enhance the semantic information of point cloud data by projecting image semantic features onto foreground points of LiDAR data. These methods include PointPainting [32] and PointAugmenting [66], where the former projects 2D semantic segmentation results onto point clouds, and the latter annotates point clouds using deep CNN features. This approach performs well in complex scenarios because it integrates rich semantic information from images directly into the geometric structure of LiDAR point clouds.

Intermediate-level fusion methods have received increasing attention in recent years. For example, ContinuousFusion [67] and DeepFusion [34] perform fusion at the deep feature level, enhancing cross-modal interaction by sharing information between 2D and 3D backbone networks. Xu et al. [53] proposed the FusionPainting framework to fuse 2D RGB images and 3D point clouds at the semantic level, enhancing 3D object detection. However, it did not fully utilize LiDAR depth information. BEVFusion improves the accuracy and robustness of multi-modal fusion by projecting image features onto the BEV space and integrating them with LiDAR data. Unlike BEVFusion, which relies on purely image-based depth estimation to construct BEV representations, LGMMfusion uses LiDAR depth to supervise the transformation, reducing spatial ambiguity. DeepFusion performs late-stage feature blending with limited geometric alignment, whereas our adaptive cross-attention explicitly models point-pixel interactions using deformable offsets. Beyond the field of autonomous driving, recent efforts in environmental modeling have also demonstrated the importance of multi-source data fusion and interpretability in deep learning. For example, Wang et al. [54] proposed a secondary modal decomposition ensemble model for groundwater prediction, while Li et al. [55] developed an interpretable deep learning framework for water quality analysis. These works underscore the cross-domain relevance of integrating heterogeneous data for robust and explainable prediction.

Feature extraction

For each input modality, we employ respective encoder networks to generate the corresponding feature maps. This section describes the feature extraction process for both raw LiDAR and image data, which are the foundation for subsequent processing in our framework.

LiDAR feature extraction.

For LiDAR point cloud data, we begin by converting the raw point cloud into a 3D voxel grid through voxelization. This step discretizes the continuous point cloud into a regular grid, making it suitable for processing by convolutional networks. To extract spatial features from the voxel grid, we utilize a Sparse Convolutional Network, which efficiently handles the sparsity of LiDAR data. The sparse convolutional layers focus on the non-empty voxels, enabling the network to capture essential structural information while maintaining computational efficiency. After passing through the encoder, the BEV feature map for the LiDAR data is denoted as , where X and Y represent the width and height of the BEV perspective, respectively, and D denotes the depth or the number of channels in the feature map. The LiDAR encoder contains four 3D sparse convolutional layers with a kernel size of at each stage. These layers progressively reduce the spatial resolution while extracting high-level geometric features from the voxelized point cloud.

The mathematical representation is as follows:

(1)

where represents the raw LiDAR point cloud data, with N being the number of points in the cloud. The LiDAR encoder, , processes the input data to yield the BEV feature map , which captures essential 3D spatial features for downstream tasks.

Image feature extraction.

For image data, we leverage the Swin Transformer to extract high-level visual features. The Swin Transformer, a hierarchical vision transformer, is particularly effective in capturing long-range dependencies within the image, which is crucial for understanding complex scenes. The image encoder performs feature extraction through a series of convolutional layers, followed by further processing via a Feature Pyramid Network (FPN). The FPN enhances the ability to capture multi-scale image features by constructing a pyramidal representation, where each level corresponds to a different scale of the image. This multi-scale feature extraction enables the model to better handle objects of varying sizes and locations within the scene. The feature map at each scale is represented as , where H_l and W_l denote the height and width of the feature map at the l-th scale, and C is the number of channels, which corresponds to the dimensionality of the feature vector at that scale.

The corresponding equation is as follows:

(2)

where is the input raw image, with H and W being the height and width of the image, respectively. The image encoder, , processes the raw image to produce the multi-scale feature map I_l, which captures visual information at different spatial resolutions. These multi-scale features are essential for detecting objects at various scales and locations in the image.

The integration of LiDAR and image feature extraction modules ensures that complementary spatial and visual information from both modalities is captured effectively. The subsequent fusion of these features allows the model to leverage the strengths of both sensors, enhancing the overall performance of 3D object detection.

Multi-head self-cross attention mechanism block

In this section, we introduce the Multi-Head Self & Cross Attention Mechanism Block(MHS-CAMB), which plays a pivotal role in generating high-quality BEV features from both LiDAR and image data. The primary goal of this block is to fuse LiDAR and image features in a multi-scale and attention-driven manner, ensuring the preservation of essential information. The key components of this module are the Multi-Head Multi-Scale Self-Attention Mechanism (MHMS-SA), Multi-Head Adaptive Cross-Attention Mechanism (MHA-CA), and the FPN for hierarchical feature refinement. Additionally, we employ additional addition and normalization layers to ensure proper alignment and feature normalization.

Multi-head multi-scale self-attention.

The LiDAR BEV features and BEV queries are processed using the MHMS-SA, as shown in Fig 2. This mechanism is designed to capture features across multiple scales by utilizing deformable attention, which ensures that both high- and low-level features are preserved. While standard Transformer self-attention computes dense pairwise interactions within a flat token sequence, our MHMS-SA module extends this idea to BEV-space feature maps by leveraging multiscale deformable attention. It samples spatially sparse but semantically rich features from both the BEV query and LiDAR feature map across multiple resolutions. This enables MHMS-SA to efficiently aggregate fine-grained spatial details and contextual cues from large receptive fields. Such a design contrasts with traditional Transformer architectures, which require either downsampling or linear projection to approximate multiscale context, often at the cost of spatial precision. By directly attending to multi-resolution feature spaces, MHMS-SA improves detection of objects with varying sizes and spatial sparsity — a key advantage in autonomous driving perception.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Multi-head multi-scale self-cross attention mechanism block.

https://doi.org/10.1371/journal.pone.0331195.g002

For embedded point features, we have the following.

(3)

where Q_p represents the sampled points in the BEV grid, Q represents the BEV Query, p is the positional information of the sampled points, and V is the feature set, which can be either the BEV queries or the LiDAR BEV features.

The deformable attention mechanism is expressed as:

(4)

where and W_mk are the weight matrices for the query and key, respectively; is the positional embedding; is the position offset; and C represents the feature dimension. This deformable attention mechanism dynamically adjusts the focus on different parts of the feature space, ensuring that spatial relationships are effectively captured across various scales.

Multi-head adaptive cross-attention.

Next, we apply the MHA-CA to align features from the LiDAR and image modalities, as shown in Fig 3. The LiDAR BEV features guide the generation of BEV features from the image. The cross-attention mechanism ensures that the generated image BEV features are spatially aligned with the LiDAR BEV features, as shown in Fig 4.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Multi-head adaptive cross-attention block.

https://doi.org/10.1371/journal.pone.0331195.g003

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. Within the BEV coordinate system, we construct a three-dimensional grid comprising sampling points.

The illustrations depict the projections of these 3D grid points onto the 2D image plane as viewed from various angles.

https://doi.org/10.1371/journal.pone.0331195.g004

The mathematical formulation of this process is:

(5)

where denotes the set of reference points associated with specific image perspectives, p_ij is the positional embedding for the reference points, and F_i is the feature processed by the self-attention mechanism.

The MSDeformAttn function computes deformable attention across multiple scales as follows:

(6)

where X_Amlqk represents the alignment weights, and and W_m are the learned weight matrices for the cross-attention operation. is the positional encoding for the reference points, is the position offset, and C represents the feature dimension. This cross-attention mechanism ensures effective alignment of LiDAR and image features, maintaining spatial consistency across modalities.

Feature pyramid network.

After the multi-head attention operations, the features undergo refinement through a FPN. The FPN generates multi-scale features by progressively refining them through a series of addition and normalization layers. This process helps capture both high-level semantic information and low-level details, making the features more suitable for downstream tasks, such as object detection.

(7)

By structuring the attention mechanisms and incorporating key refinements through the FPN, we ensure that the model produces high-quality BEV features that are both spatially coherent and semantically rich. Compared to prior transformer-based fusion methods such as FUTR3D, which rely on global cross-attention to inject image features into 3D object queries, our adaptive cross-attention module introduces several key innovations. First, we use LiDAR-derived geometric priors to guide the sampling positions, improving spatial alignment between modalities. Second, our module applies a multi-head multi-scale deformable attention mechanism, which dynamically selects relevant features from multiple image scales based on BEV query positions. Finally, we incorporate a learnable offset refinement process that adapts sampling points to the underlying scene geometry. These improvements allow LGMMfusion to maintain better consistency in spatial feature correspondence, especially under sparse or occluded input conditions.

Fusion block

In this section, we introduce the fusion block, which is designed to effectively combine the LiDAR BEV features with the image BEV features to generate the final fused BEV features. This module plays a crucial role in integrating multimodal information, thereby enhancing the performance of multimodal perception tasks. The fusion block consists of the following key steps:

Dynamic weighted fusion.

Firstly, the BEV features from LiDAR and image are fused using a dynamic weighted mechanism. This fusion process is carried out through a weighted concatenation operation, resulting in the fused feature , which is computed as follows:

(8)

Here, and are dynamic weight coefficients that control the importance of the LiDAR and image features during the fusion process. These coefficients allow for flexible and adaptive adjustment of the contributions from each modality depending on the context, ensuring that the most relevant features are emphasized in the final fused representation.

Deformable convolution and deep fusion network.

After the fusion step, the concatenated feature undergoes deformable convolution (DeformConv), which adjusts the spatial alignment of the features. Deformable convolutions enable the network to sample features more flexibly in space, allowing for better spatial alignment and improved feature representation, especially in the presence of complex object shapes and spatial variations.

Following the deformable convolution, the fused features are further processed using a Multi-Layer Perceptron (MLP) network for deep fusion. The MLP network captures the non-linear relationships between the features and enhances their representational capacity. Specifically, the fused feature undergoes convolution (Conv) and batch normalization (BN), followed by a ReLU activation function, to obtain the final fused feature:

(9)

This deep fusion network not only refines the feature representations but also improves the model’s ability to capture complex interactions between LiDAR and image features, thus enhancing the overall perception capability.

SECOND algorithm and BiFPN optimization.

The resulting fused feature is then passed through the SECOND algorithm for further optimization. The SECOND algorithm utilizes sparse convolutions to improve the efficiency of point cloud processing, optimizing the spatial structure of the fused features. This step allows the model to process large-scale point clouds more efficiently while preserving the important spatial details. The features are then further enhanced through the Bidirectional Feature Pyramid Network (BiFPN), which is capable of fusing multi-scale features more effectively. BiFPN enables the model to focus on features at different spatial resolutions and progressively refine them for more accurate predictions. The final fused feature is computed as follows:

(10)

The integration of BiFPN significantly optimizes the fusion process across multiple scales, ensuring that both fine-grained details and broader context are incorporated into the final feature representation. This step not only enhances feature refinement but also contributes to the robustness and accuracy of the multimodal perception system. Through these steps, the fusion block effectively combines the LiDAR BEV features and image BEV features, generating a more accurate and efficient final BEV feature. The dynamic weighted fusion allows for flexible adjustment of feature importance, while deformable convolution and the deep fusion network enhance feature representation. Finally, the SECOND algorithm and BiFPN work together to refine the fused features, resulting in improved multi-modal perception performance. This multi-stage fusion process ensures that both the spatial and semantic aspects of the multimodal data are effectively integrated, leading to significant improvements in the accuracy and efficiency of the perception system.

Detection head and loss function

Building on the detection head design in TransFusion-L, we propose a similar yet modified detection head and loss function, capable of handling both LiDAR and image data. The following outlines the design approach.

Loss function.

We design a loss function similar to that of TransFusion-L, addressing issues such as class imbalance, bounding box localization, and center heatmap loss.

Total Loss Function:

(11)

We use the Smooth L1 loss (Huber loss) instead of the L1 loss, providing more stability when handling outliers and smoother training progress. Bounding Box Regression Loss:

(12)

where B_i is the ground truth bounding box and is the predicted bounding box.

To address class imbalance, we replace focal loss with balanced cross-entropy loss:

(13)

where w_i is the class weight factor to balance class distribution.

We use a Gaussian heatmap loss with IoU weighting to impose a stronger penalty for localization errors, especially for highly overlapping objects. Heatmap Loss (Gaussian Heatmap + IoU-weighted):

(14)

where h_i is the ground truth heatmap, and is the predicted heatmap.

Given the multimodal nature of the data, semantic loss ensures that detected objects align with predefined semantic categories (e.g., vehicles, pedestrians). This loss function serves as an additional term in the total loss. Semantic Loss:

(15)

where p_i is the predicted probability of the object’s semantic category.

Dataset

nuScenes dataset is a comprehensive, large-scale dataset developed by Motional for autonomous driving research [68]. It features a diverse collection of 1,000 driving scenes from two cities, Boston and Singapore, renowned for their challenging traffic conditions. Each scene is 20 seconds long and showcases various driving maneuvers, traffic situations, and unexpected events, reflecting the complexities of urban driving. The dataset includes rich multi-modal sensor data from a full sensor suite, consisting of 6 cameras, 1 LiDAR, 5 radars, GPS, and IMU.

nuScenes offers 23 object categories annotated with accurate 3D bounding boxes at 2Hz, making it ideal for object detection and tracking tasks. In addition, object attributes such as visibility, activity, and pose are annotated, enhancing its utility for advanced perception tasks. The dataset contains approximately 1.4 million images from cameras, 390,000 LIDAR sweeps, 1.4 million RADAR sweeps, and 1.4 million object bounding boxes in 40,000 keyframes.

Evaluation criteria

The LGMMfusion detection task involves detecting 10 object categories, with each object annotated with 3D bounding boxes, attributes, and velocities. To assess the accuracy of the detection, matching is performed by calculating the distance d between the 2D center of the predicted 3D bounding box on the ground plane and the actual ground truth. Predictions with recall or precision lower than 10 % are filtered out to ensure only relevant matches are considered. The Average Precision (AP) is then calculated as the normalized area under the precision-recall curve.The overall mean Average Precision (mAP) is defined as:

(16)

where: C denotes the set of predicted classes, D represents the set of distance thresholds between the predicted and true centers on the ground plane, is the Average Precision for a class c and distance threshold d. For our evaluation, we adopt the following distance thresholds .

Additionally, five key metrics for True Positives (TP) are computed during the matching process: Average Translation Error (ATE): Measures the translation accuracy of the detected object. Average Scale Error (ASE): Evaluates the accuracy of the object scale prediction. Average Orientation Error (AOE): Assesses the rotational accuracy of the detected object. Average Velocity Error (AVE): Quantifies the error in velocity prediction. Average Attribute Error (AAE): Measures how accurately the object attributes, such as visibility or activity, are predicted.Each of these metrics is calculated based on a 2-meter center distance threshold, ensuring robust performance metrics for localization and object behavior prediction.To summarize the detection performance, we compute the mean True Positive rate (mTP) as:

(17)

where TP_c represents the number of True Positives for class c. Finally, the nuScenes detection score Detection Score (NDS), which serves as the integrated metric combining both detection performance and quality, is calculated as follows:

(18)

This score reflects the overall effectiveness of the detection system by balancing both the precision of the object detection (via mAP) and the quality of the object localization (via mTP). By leveraging these evaluation metrics, LGMMfusion provides a comprehensive and reliable performance measure for autonomous driving detection tasks.

Experimental settings

Model Setup: The LGMMfusion network is built upon the open-source 3D object detection framework mmdetection3D. Our framework is implemented using PyTorch, which is optimized for training on GPUs. For image feature extraction, we use a pre-trained Swin-Tiny model, which has been pre-trained on the ImageNet dataset. To optimize training efficiency and reduce GPU memory usage, we freeze the weights of the pre-trained model during the entire training process. For LiDAR feature extraction, the point cloud is voxelized, and sparse convolutions are used to extract the features from the voxelized data.

In the process of generating the BEV image from the LiDAR data, we utilize 6 layers of MHMSSA and MHACA. Additionally, we account for the effects of data augmentation on the transformation matrix . This transformation matrix is updated based on the augmentation matrices and . The input image size is set to 256x704 pixels, and the voxel size is defined as . The BEV feature map resolution is set to based on the voxel grid, with each voxel representing 0.075 meters in the X and Y dimensions and 0.2 meters in the Z dimension. During image BEV feature generation, we adopt deformable attention with 8 sampling points per reference and 4 spatial scales, which aligns with the structure of our multi-head adaptive cross-attention mechanism.

Training Details: The LGMMfusion network is trained using PyTorch 2.1.0 with CUDA 11.8 and cuDNN 8.6. For end-to-end training, we leverage a distributed setup with 8 NVIDIA A100 GPUs. In the first phase, we train the LiDAR branch of the network for 25 epochs using the AdamW optimizer with a learning rate of 0.00005. In the second phase, the image branch is introduced, and training continues for another 12 epochs, using the same AdamW optimizer with a learning rate of 0.00005. During inference, we do not use test-time augmentation or multi-model ensembling to ensure consistent results across different experiments.

Experimental results and analysis

We conducted extensive experiments to evaluate the performance of our proposed method, LGMMfusion, and compared it with several state-of-the-art approaches. These include camera-based methods FCOS3D [70] , PGD [71] and BEVFormer [49], LiDAR-based methods such as PointPillars [69], SECOND [43], and CenterPoint [72], as well as LiDAR-camera fusion models like CFF [74], UVTR [20], mmFusion [78], and FUTR3D [64]. The results on the nuScenes validation set are presented in Table 1.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Results on nuScenes val set. The modalities are Camera (C), LiDAR (L).

https://doi.org/10.1371/journal.pone.0331195.t001

The experimental results demonstrate that LGMMfusion achieves significant performance improvements over both pure camera-based and multi-modal fusion methods. Specifically, compared to pure camera-based approaches, LGMMfusion outperforms FCOS3D and PGD by a large margin. FCOS3D and PGD exhibit lower detection accuracy, with mAP scores of 0.343 and 0.369, respectively, significantly trailing behind LGMMfusion’s 0.673. This discrepancy highlights the inherent limitations of monocular 3D object detection, particularly in estimating accurate depth and spatial localization. Additionally, their lower NDS values (0.415 and 0.428) further confirm the challenge of achieving robust detection using camera-only methods, reinforcing the necessity of integrating LiDAR data to improve overall perception accuracy. Furthermore, compared to BEVFormer, a more advanced camera-based method, LGMMfusion still demonstrates substantial advantages. Our approach achieves a relative improvement of 25.7% in mAP and 19. 4% in NDS, together with notable reductions in mATE and mASE. These improvements emphasize the effectiveness of leveraging both LiDAR and camera data for enhanced detection performance. The ability of LGMMfusion to incorporate precise geometric depth information from LiDAR while preserving rich semantic cues from camera data enables it to achieve superior accuracy, particularly in challenging scenarios with occlusions and distant objects.

Comparing LGMMfusion with LiDAR-based methods like PointPillars, SECOND, and CenterPoint, our model demonstrates substantial improvements in both mAP and NDS. LGMMfusion achieves an mAP improvement of 15.7%, 14.5%, and 14.1% over PointPillars, SECOND, and CenterPoint, respectively. In terms of NDS, our model surpasses these methods by 9.0%, 8.2%, and 6.2%, respectively, demonstrating the benefits of multi-modal fusion over LiDAR-only approaches.

When compared with existing LiDAR-camera fusion models such as CFF, UVTR, mmFusion, and FUTR3D, LGMMfusion also achieves state-of-the-art performance. Our method surpasses CFF and UVTR by 2.2% in mAP, mmFusion by 1.9%, and FUTR3D by 0.7%. Furthermore, in terms of NDS, LGMMfusion achieves an improvement of 2.2% over CFF and UVTR, 1.4% over mmFusion, and 2.8% over FUTR3D.

One notable observation is that despite being purely camera-based, BEVFormer demonstrates a competitive performance compared to other vision-only methods, with an mAP of 0.416 and an NDS of 0.517. However, its performance still lags significantly behind multi-modal fusion methods, particularly in localization accuracy, as indicated by its higher mATE value (0.672). This further supports the argument that while advanced camera-based approaches can achieve moderate accuracy, the absence of depth-aware LiDAR information remains a critical bottleneck.

In terms of error metrics, LGMMfusion reduces mATE, mASE, and mAAE compared to other methods, demonstrating improved localization accuracy and robustness. In particular, our method achieves a 40.6% reduction in mATE compared to BEVFormer, indicating significantly enhanced depth estimation and object localization.

Overall, the experimental results on the nuScenes dataset confirm that LGMMfusion effectively combines LiDAR and camera data, resulting in a substantial performance boost in 3D object detection. Our method consistently outperforms both LiDAR-only and camera-only approaches across multiple evaluation metrics, reinforcing the advantages of multi-modal fusion.

To further emphasize the timeliness of our approach, we compare LGMMfusion against several recently proposed multimodal 3D detection frameworks, including FUTR3D [64], UVTR [20], and mmFusion [78]. As shown in Table 2, LGMMfusion outperforms these methods in both mAP and NDS. In particular, it improves small object detection categories such as pedestrians and traffic cones by 2.5–5.0%, highlighting the effectiveness of our LiDAR-guided BEV feature alignment. Unlike FUTR3D, which fuses image features at the proposal level, our method performs early spatial alignment at the BEV level, enabling more coherent feature representation for dense urban scenes. These comparisons confirm that LGMMfusion remains competitive with and complementary to the latest state-of-the-art methods in multimodal perception.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Results on the nuScenes validation set. The results are compared across different methods using LiDAR (L) and Camera (C) modalities(Mod). The performance is evaluated both overall (mAP, NDS) and at the per-class level [75]. The classes include Car, Truck(Tru), Construction Vehicle (C.V.), Bus, Trailer(Tra), Barrier(Bar), Motorcycle (Motor), Bike, Pedestrian (Ped.), and Traffic Cone (T.C.).   Small object categories include pedestrian, traffic cone, and bicycle, which show the most notable mAP improvements.

https://doi.org/10.1371/journal.pone.0331195.t002

In addition to detection performance, we also report the computational complexity of LGMMfusion. The model requires approximately 125 GFLOPs per frame and achieves an inference speed of 12.3 FPS on an NVIDIA A100 GPU. In comparison, CenterPoint consumes 92 GFLOPs and runs at 19.4 FPS under the same conditions. While LGMMfusion introduces higher computational cost due to image BEV generation and attention mechanisms, it achieves significantly better detection performance, especially on small and occluded objects. This trade-off is acceptable for high-accuracy perception scenarios such as autonomous driving.

Fig 5 visualizes the detection results in the BEV framework. Our approach effectively identifies objects even in low-density point cloud regions. This improvement is primarily due to the synergistic integration of LiDAR and image features, where high-resolution semantic details from the image modality compensate for sparse LiDAR information, particularly for distant and small-scale objects.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. This figure shows an example of 3D annotation, each row represents a scene.

The first row is a tunnel during the day, and the second row is a tunnel at night.

https://doi.org/10.1371/journal.pone.0331195.g005

Table 2 presents the 3D detection results on the nuScenes validation set, where we compare the performance of LGMMfusion with several existing methods, including CenterPoint, CFF, mmFUSION, and FUTR3D. The evaluation metrics include mean Average Precision (mAP) and NuScenes Detection Score (NDS), as well as per-class performance for several object categories.

Overall, LGMMfusion achieves competitive performance with an mAP of 67.3% and NDS of 71.1%, showing noticeable improvements over CenterPoint and FUTR3D in several categories. For instance, the Traffic Cone (T.C.) class shows a significant improvement of 5% over mmFUSION, while the Bus category shows a modest gain of 4.1% compared to FUTR3D. These results indicate that our method performs well in detecting smaller and more challenging objects.In terms of class-specific performance, LGMMfusion also shows reasonable gains in categories like Pedestrian (Ped.) and Motorcycle (Motor.), with mAP improvements of approximately 3.0% and14.4% over CenterPoint, respectively. These gains suggest that LGMMfusion benefits from its fusion of LiDAR and camera modalities, particularly in handling small-sized objects where traditional methods may struggle.

In summary, LGMMfusion demonstrates solid improvements, particularly in detecting smaller objects like Pedestrian and Traffic Cone, but the enhancements are more moderate in larger object categories. The results validate the effectiveness of our multi-modal fusion approach in enhancing the detection of various object scales without drastic performance trade-offs.

Ablation study

In this section, we present the results of our ablation study on the nuScenes validation set, as shown in Table Table 3. The purpose of this experiment was to investigate the impact of different components of the LGMMfusion model. We evaluated four model variants: a baseline using LiDAR data only, and three additional configurations incorporating camera data and variations in the Image BEV processing.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Extended ablation study results on the nuScenes validation set. This table compares the performance of LGMMfusion variants, including the impact of Image BEV (I-BEV), BEV Query (BEV-Q), attention structure modules (MHMS-SA and MHA-CA), and attention parameters (number of heads H and scales S). I-BEV variations involve the use of BatchNorm (BN) and ReLU activations.

https://doi.org/10.1371/journal.pone.0331195.t003

As the baseline, the LiDAR-only model achieved an mAP of 63.4 and an NDS of 68.8. This configuration serves as a reference point for assessing the performance improvements introduced by incorporating additional data modalities and processing strategies. When the camera modality was introduced along with BatchNorm (BN) processing, the model exhibited a notable improvement, with mAP increasing to 65.1 and NDS reaching 69.2. This demonstrates that leveraging camera information, even with basic normalization techniques, enhances detection accuracy and object localization by enriching semantic context and structural details.Further performance gains were observed when the BEV Query (BEV-Q) mechanism was introduced, leading to an mAP increase to 66.3 and an NDS improvement to 70.1. The 1.2% rise in mAP over the previous configuration indicates that BEV Query effectively refines the representation of fused LiDAR and camera features, enabling more accurate object detection in 3D space. This suggests that dynamically querying BEV features facilitates better spatial feature alignment, improving object representation across different sensor modalities. The final configuration, which incorporates ReLU activation into Image BEV processing, yielded the highest performance, with mAP increasing to 67.3 and NDS reaching 71.1. This improvement highlights the crucial role of non-linearity in refining feature transformations. ReLU enhances the expressiveness of extracted image features, allowing for more robust feature interactions between LiDAR and camera data. Furthermore, the noticeable reductions in mATE and mAOE indicate improved localization precision and orientation estimation, confirming the positive impact of this enhancement.

In summary, each modification introduced in this study contributed to progressive improvements in detection performance. The integration of camera data with LiDAR provided a solid foundation for multimodal fusion, while the addition of BEV Query and ReLU further refined feature alignment and representation. These results emphasize the effectiveness of our fusion strategy in LGMMfusion and demonstrate the critical role of both advanced processing techniques and multimodal data integration in optimizing 3D object detection.

To further analyze the contribution of individual modules, we performed additional ablation experiments by disabling the MHMS-SA and MHA-CA blocks separately. Removing the MHMS-SA module reduced mAP from 67.3% to 66.2%, while disabling the MHA-CA module resulted in an mAP of 65.9%. These results confirm that both components play critical roles in enhancing feature representation and cross-modal alignment. We also conducted a sensitivity analysis on the number of attention heads (H) and the number of spatial scales (S) used in the deformable attention mechanisms. The model achieves optimal performance when H = 8 and S = 4. Increasing these values beyond this point yields negligible improvement or mild overfitting, whereas reducing them leads to degraded detection accuracy. Detailed results are included in Table 3.