3.2.1 3D-to-2D projection with Gaussian splatting‌‌.

To leverage 2D convolutional networks on irregular point clouds, we first project the 3D points onto a 2D grid, creating a pseudo-image. A naive projection can cause aliasing and information loss, especially in sparse regions. We address this by applying Gaussian splatting as a pre-processing step, which smooths the point cloud and distributes each point’s influence to its neighbors before projection. For any point and a neighboring point p_j = (x_j, y_j, z_j), we define the Gaussian weight (influence) of p_k on p_j as:

(1)

where is the standard deviation of the Gaussian kernel controlling the spatial extent of the influence. This weight decays exponentially with Euclidean distance, so closer points have a larger mutual influence. Intuitively, Gaussian splatting performs a locally weighted smoothing of the point cloud: each point “spreads” a fraction of its value (position or features) to nearby points, akin to distributing mass in a continuous density map. This process reduces noise and fills small gaps by ensuring that isolated or sparse points contribute to the representation of their vicinity, thereby enhancing local continuity.

Using the weights from (1), we adjust each point’s position toward its neighbors. For a given point p_k and one of its neighbors p_j, the neighbor can be shifted slightly toward p_k by:

(2)

where is the updated neighbor position. In practice, we iteratively or simultaneously apply such shifts for all points in a local neighborhood, effectively smoothing the point cloud geometry. The influence of p_k on p_j is strongest when p_j lies very close to p_k, and diminishes for distant points. As a result, Gaussian splatting preserves fine local structures while mitigating outliers: points within a tight cluster move closer together (denoising small isolated noise), and small gaps are bridged by intermediate influence, but large-scale geometry remains unchanged since distant points have negligible effect on each other. After Gaussian splatting, the point distribution is more uniform and continuous, which is beneficial for the subsequent projection.

3.2.2 Quantized projection to image plane.

Once the point cloud has been smoothed, we project the 3D points onto a 2D plane to generate a rasterized representation. Given an appropriate viewpoint (e.g., an overhead orthographic view for aerial data or a perspective view with known camera intrinsics/extrinsics), we define a projection function. For simplicity, assuming an orthonormal projection, we map coordinates by quantizing the x and y values of each point to integer pixel coordinates:

(3)

where (u_k, v_k) are the 2D pixel coordinates of point p_k on the image plane, and s is a quantization factor (grid resolution) that determines the size of each pixel. This operation effectively bins points into pixels: all points falling into the same bin are mapped to the same image pixel. In a more general camera model, would incorporate perspective projection equations using calibration parameters, but the end result is the same—a set of image coordinates corresponding to the 3D points. We retain the association between each point p_k and its pixel (u_k, v_k) for later fusion. The projection compresses the 3D structure onto a 2D grid while, thanks to Gaussian splatting, preserving neighborhood relationships. Specifically, the Gaussian smoothing ensures that if a 3D structure is somewhat sparse, its projection will still carry a continuous intensity or feature distribution rather than isolated pixels, thereby reducing information loss during quantization.

The projection yields an initial 2D image I (with channels for accumulated color or density) of size H × W pixels covering the area of interest. Each pixel in I encodes the presence (and possibly aggregated color/features) of points in the corresponding 3D column of space. This forms the input to the image processing stage of the RI branch.

3.2.3 Multi-scale CNN feature extraction.

With the 2D projection I in hand, we employ a 2D convolutional neural network to extract high-level features that capture textures, edges, and contextual cues from the remote sensing image. We denote the image feature extraction as:

(4)

where is the resulting feature map (with C₁ channels) produced by the CNN . This CNN can be a standard encoder (e.g., ResNet or U-Net backbone) that progressively extracts features from low-level (edges, textures) to high-level (object-level context). We adopt a multi-scale feature extraction strategy to ensure both local details and global contexts are captured. In practice, this can be implemented by a feature pyramid network or by using convolution layers with different receptive fields. We obtain feature maps at multiple scales (levels of resolution) and combine them to form a comprehensive representation:

(5)

where is the feature map extracted at scale (or level) ℓ, and is a learned weight reflecting the importance of that scale. The summation in (5) indicates a weighted fusion (or it could be a concatenation followed by a 1x1 convolution) of multi-scale feature maps, yielding a final 2D feature representation that contains information from coarse (contextual) to fine (detailed) levels. By doing so, the RI branch ensures that features like building edges or tree textures (fine details) as well as broader layout cues (e.g., a cluster of pixels forming a large roof or road, giving context) are both preserved. The output of this branch is a dense feature map covering the entire projected scene. This Remote-Image branch output will later be aligned and fused with the point cloud branch output. Crucially, the RI branch provides rich color and texture cues that complement the geometric features from the point cloud branch, helping to distinguish objects that might be geometrically similar but visually different (for instance, a tree vs. a lamp post might have similar shapes but very different image textures).

3.3.1 Geometric clustering for regional features.

Instead of treating every point in isolation, we first organize the point cloud into meaningful clusters that reflect local surfaces or objects. Clustering reduces data complexity and provides a structural context for each point. We employ a density-based clustering (DBSCAN) to group points that are spatially close and densely distributed. Formally, for DBSCAN we define a neighborhood radius and minimum points m. A point p_i is considered a core point if it has at least m neighbors within distance . DBSCAN then defines an equivalence relation on points: p_i ∼ p_j if there exists a chain of points such that each consecutive pair lies within of each other. All points related by ∼ form a cluster C_k. This identifies contiguous high-density regions (e.g., points on the same facade or ground patch). To capture larger-scale groupings, we also construct a graph G=(V,E) over clusters or points, where each node represents a point (or an initial cluster) and edges connect points that are in proximity or have similar attributes (color, normal, etc.) even if they belong to different DBSCAN clusters. For example, we add an edge between p_i and p_j if (distance threshold) or if their feature similarity (e.g., normal vector difference or color difference) is above a threshold . We can then apply graph-based clustering (such as spectral clustering or community detection on G) to merge the DBSCAN clusters into larger super-clusters or supervoxels . The result of this two-stage clustering is a set of clusters (ranging from small, dense regions to broader structures). We assign each point p_i a cluster label indicating which cluster it belongs to. We also compute a cluster-level feature descriptor for each cluster C_k, for instance by averaging or pooling the point features in that cluster:

(6)

where f(p_i) could be the initial feature of point p_i (such as a positional encoding or color) or a learned intermediate feature. These cluster descriptors provide a compact representation of region-wise properties. The clustering step benefits segmentation by capturing regional context: points within the same cluster likely belong to the same object or surface (hence share a label), so the model can use the cluster identity as a strong cue for segmentation consistency. Moreover, clustering helps focus the later stages of the network on meaningful groups rather than isolated points, improving efficiency and robustness to noise (outliers may fall into small clusters or remain unclustered and can be handled separately).

3.3.2 Local feature encoding (PointNet++).

It is important to note that PointNet++ is adopted here in a specific and limited role: it serves solely as a local neighborhood feature encoder, not as the primary backbone of the network. Global contextual modeling is explicitly delegated to the dedicated self-attention Transformer in Module 3, while relational refinement is handled by the GCN in Module 4. Replacing PointNet++ with a globally-aware architecture such as Point Transformer [24] or PointNeXt would introduce functional redundancy with the downstream Transformer module, increasing model complexity without commensurate benefit. PointNet++ is therefore deliberately chosen as a lightweight and effective component for this local encoding stage, consistent with our design goal of maintaining inference efficiency. Next, we extract point-wise features that capture fine-grained local geometry. Drawing inspiration from PointNet++’s set abstraction, we consider each point p_i and its neighborhood N(i) (for example, all points within radius r of p_i, or the k nearest neighbors). We then apply a small point-based neural network (MLP) on each neighbor’s relative position and initial feature (e.g., relative coordinates and color) and aggregate these to form a local descriptor for p_i. Formally, let be an MLP that maps a concatenation of a neighbor’s relative coordinates and features to a latent representation. We compute:

(7)

where denotes concatenation of the geometric offset p_j − p_i and the neighbor’s feature f(p_j) (which initially could be just color or height, and later can be replaced by intermediate learned features), and the operator is applied element-wise to the set of neighbor feature outputs to yield an invariant summary (one could also use average pooling). The result is a fixed-length feature vector encoding the shape of the local neighborhood around p_i (e.g., flat surface vs. edge vs. corner) and the distribution of any attributes. This is analogous to the local feature extraction in PointNet++  which captures fine geometry like curvature or roughness around each point. By computing , the network gains an understanding of small-scale structures, which is crucial for distinguishing adjacent classes (for instance, ground vs. curb vs. wall might differ in the curvature or normal variation in their local neighborhoods).

3.3.3 Global context encoding (self-attention transformer).

While local features are important, points in large-scale scenes can be far apart yet related (e.g., two walls on opposite sides of a street are both walls and part of the same street scene). To capture long-range dependencies and global context, we feed the point features into a self-attention module inspired by transformer architectures. We treat the set of point features as a sequence of “tokens.” For each point i, we compute query, key, and value vectors via learnable linear projections of the local feature:

(8)

where are projection matrices (with din the dimension of h^locali and d the chosen dimensionality of the queries/keys). The self-attention mechanism allows each point i to attend to all other points j through the compatibility of their query and key. We compute attention weights and updated features as:

(9)

Here represents the attention weight that point i assigns to point j; it is essentially a learned affinity between points i and j normalized across all j. The updated feature is a weighted sum of value vectors from all points, so it integrates information from the entire point cloud, with more focus on points that are deemed relevant to i. In practice, we use multi-head self-attention: the projection in (8) and computation in (9) are done H times (with independent for head and smaller d per head), and the results are concatenated to form a rich global feature for each point. The transformer’s self-attention endows each point’s representation with awareness of the overall scene layout and long-range interactions. For example, a point on a car will receive contextual information from other points on the same car (to understand the extent of that object) and even from points on the road underneath or buildings nearby, as the model learns to attend to relevant contexts for segmentation. This global feature helps disambiguate cases where local geometry alone is insufficient (many objects might have planar surfaces locally, but their global arrangement or co-occurrence helps identify what they are).

3.3.4 Relational refinement with graph convolution.

After obtaining global context features, we further refine the point representations by incorporating local relational structure explicitly, using Graph Convolutional Networks (GCNs). While the self-attention treated all point pairs with equal potential, a GCN can enforce that only points with actual connections (e.g., spatial neighbors or within the same cluster) exchange information, thus reinforcing locality and smoothing within object regions. We construct an adjacency relation A_ij for points, for example A_ij = 1 if point (a neighbor in the point cloud within radius or KNN) or if c(j)=c(i) (points belong to the same cluster from step 1), and A_ij = 0 otherwise. A simple first-order graph convolution update for each point i is then:

(10)

where W₀, W₁ are learnable weight matrices and is a nonlinear activation (e.g., ReLU). In this formulation, each point i updates its feature by combining its current feature with an average of the features of its neighbors . This is essentially a graph smoothing or averaging that promotes similarity among connected points. More advanced GCN variants could weight each neighbor in the sum by a function of and or by edge attributes, but the essence remains: points that are spatially or semantically connected (like belonging to the same cluster or object) will align their features. This graph convolutional refinement encourages consistency: if a point is on a planar roof and its neighbors are also on that roof, their features after GCN will become more similar, making it easier for the classifier to assign them the same label. Conversely, at object boundaries, neighbors might belong to different clusters or be excluded from the adjacency, allowing discontinuities to remain where appropriate. After the GCN step, we obtain the refined feature for each point p_i. At this stage, each encodes multi-scale information: (i) the local neighborhood geometry (from PointNet++), (ii) the global context (from transformer attention), and (iii) the relational context enforcing that points on the same structure have similar representations (from GCN). We denote this entire point feature extraction process as a function ,parameters), summarizing that for each point the branch produces a C₂-dimensional feature vector.

3.3.5 Adaptive point sampling (efficiency and focus).

Processing every point in a large-scale cloud (with millions or more points) is computationally expensive and sometimes redundant. We incorporate an adaptive sampling strategy to select a subset of points that are most informative for segmentation, reducing the computational burden while maintaining performance. The idea is to reduce the number of points from N to M (M < N) for certain expensive operations, focusing on critical areas (e.g., boundaries, distinct structures) and down-sampling repetitive or homogeneous regions.

One approach is importance-based sampling. We assign each point p_i an importance score I(p_i) that reflects how much that point contributes to segmentation accuracy. The score can be hand-crafted (e.g., higher for points with large curvature, significant height difference from neighbors, or near color edges in the image) or learned (e.g., using a small network that predicts uncertainty or feature magnitude). We then choose the top M points with the highest scores. Mathematically, if is the index set of selected points (with |S| = M), we want

(11)

meaning we aim to maximize the total importance of retained points. In practice, solving (11) exactly is combinatorial, so we use greedy heuristics or thresholding on I(p_i) (e.g., pick all points above a certain importance value). This tends to keep points on edges or uncommon classes (high I) while dropping points in flat, uniformly labeled areas (low I), thus preserving segmentation-critical structures.

Another approach is reinforcement learning-based sampling. We can frame point selection as a sequential decision process where a policy chooses points to include such that a reward (e.g., segmentation accuracy or IoU on a validation set) is maximized. A neural network can be trained to output a probability of selecting each point, and a reinforcement learning algorithm (like policy gradients) updates based on the reward. For example, an agent could learn to mimic farthest point sampling (to ensure coverage) but also to focus on high-curvature areas (to ensure detail). A differentiable approximation of sampling (such as the SampleNet approach) creates a soft selection by generating new points as weighted combinations of original points:

(12)

with weights w_ij that form a sparse selection matrix (each ideally equals one of the input points if one weight is 1 and others 0). This can be trained end-to-end by backpropagating a surrogate loss that encourages picking points that lead to good segmentation. Regardless of the mechanism, the output of adaptive sampling is a reduced set that is passed through the feature extraction pipeline (or one could apply sampling at multiple stages, such as at each layer of a hierarchical set abstraction, described next). By focusing computation on the most informative points, the network can allocate more capacity to complex regions. Moreover, sampling can act as a form of data augmentation and regularization: if at each epoch a slightly different subset of points is chosen, the model is exposed to varied inputs and must rely on robust features rather than overfit to every single point. Our adaptive sampling thus strikes a balance between efficiency and accuracy, ensuring that minimal critical information is lost when reducing point density.

3.3.6 Multi-scale feature pyramid.

Urban scenes contain structures of vastly different scales, from small objects like street signs to large expanses like roads and buildings. To achieve scale-invariant segmentation, we adopt a multi-scale processing strategy for the point cloud akin to an encoder-decoder pyramid. We generate a hierarchy of point sets with decreasing resolution: P⁽⁰⁾ = P is the original set of N points, and for each level we sample a subset such that . The sampling can be random, via farthest point sampling (FPS), or using the aforementioned adaptive sampling. Points in can be thought of as representative points (or centroids) for neighborhoods in . For each representative point , we define its receptive field in the lower level as the set of points within a radius : . We then aggregate features from level ℓ to compute the feature for q at level . For example, using a PointNet-based aggregator:

(13)

where is the feature of point p at level ℓ (with h⁽⁰⁾p initialized as some function of raw coordinates and color, or after initial local feature encoding), and is an MLP that combines a point’s current feature with its relative position to the representative q. The max (or average) pooling yields the feature that represents all points in q’s neighborhood. Equation (13) is applied iteratively from up to L, yielding progressively coarser but more context-encompassing features. By level L, we might have only a handful of points (superpoints) summarizing the entire scene with very large receptive fields.

After obtaining multi-scale features, we can fuse features across scales to enrich the representation of each original point with both fine and coarse information. One approach is to propagate the coarse features back to each original point (a decoder or upsampling step). For each point in the original cloud, we find its corresponding ancestor in P⁽¹⁾ (the representative that covered p_i’s neighborhood), then that point’s ancestor in P⁽²⁾, and so on up to . We then concatenate all these features:

(14)

where denotes the feature of the ancestor of point p_i at level ℓ. In practice, a learned fusion (such as summing after projecting to a common dimension, or using attention across scales) can be applied instead of simple concatenation. The result is a multi-scale feature descriptor for point p_i that encodes information from a very local scale (level 0, possibly after the GCN refinement) up to a global scale (level L). This is highly beneficial for segmentation: small objects or details (e.g., a traffic sign) will primarily be recognized from the fine-scale component of , whereas large structures (like terrain vs. building) will be distinguished by coarse-scale components. The combination allows the network to handle scale variability: for example, what constitutes a “wall” can be a few meters or tens of meters long, but the multi-scale feature can represent both a segment of wall and the fact that it is part of a larger building. Empirically, multi-scale fusion improves the model’s ability to accurately segment both small and large objects, leading to more coherent and complete scene interpretations.

3.3.7 Point-image feature alignment for fusion.

The culmination of the point-cloud branch is a rich set of features for each point p_i. Before fusing with image features, we ensure that these 3D features are properly aligned with their corresponding image locations. Since we projected each point onto the image plane earlier (Equation (3)), we know the pixel coordinate (u_i, v_i) associated with point p_i. We leverage this correspondence to pair each point’s 3D feature with the 2D feature at the same location. Let be the 2D fused feature map from the RI branch (Eq. (5)). For each point p_i, we retrieve the image feature vector at its pixel: . Now we form an initial fused representation for point i by concatenation:

(15)

which simply attaches the 2D modality features to the 3D modality features for the same physical point. In our implementation, we typically project both and g_i into a common size and sum or concatenate them as appropriate. The key point is that by using the known projection mapping , we achieve a one-to-one point-to-pixel correspondence—each point feature knows exactly which image feature comes from the same location in the world. This eliminates any need for ambiguous nearest-neighbor matching between modalities and ensures that fusion is geometrically meaningful. For instance, a point on a tree will be paired with the exact image pixel of that tree’s canopy or trunk, so the green texture from the image is directly fused with the 3D shape from the point. This alignment sets the stage for the subsequent cross-modal fusion module, which will more deeply integrate these features. The initial concatenation could also be processed through a small network to begin mixing the modalities (e.g., a linear layer or two to produce a fused vector ), or structured in a grid (for example, forming a 2D grid of fused features if we rasterize the point cloud) to apply further convolution. In our design, we proceed with an attention-based fusion described next.

3.4.1 Segmentation head and output.

The segmentation head is a classifier that maps each fused feature F_fusion(i) to a distribution over the C semantic classes. This can be implemented by a small fully-connected network (two or three layers) or even a single linear layer followed by softmax. Formally, let denote the segmentation head mapping a fused feature to class logits. We apply it to each fused feature vector:

(22)

where is the vector of predicted scores or probabilities for the C classes for the point/pixel i. Applying S to all fused features yields , which corresponds to the semantic segmentation of the projected point cloud (in image space). Finally, these predictions are mapped back to the original points in P (since each point had a correspondence i in the fused feature set) to obtain the label predictions for each 3D point p_k. In summary, the attention-guided fusion ensures that each point’s label decision is informed by both its 3D geometric context and 2D appearance context. This design effectively leverages the complementary nature of the modalities: for example, points on a power line (thin geometry) get support from the image (distinct color against sky), while points on a road (uniform color) get support from 3D structure (flat horizontal surface spanning a large area). The end result is a more accurate and consistent segmentation of the urban scene, with fine details preserved and global context understood.

3.5.1 Information preservation loss (L_info).

Projecting a point cloud to 2D has the risk of losing some of the 3D structural information (due to occlusions or quantization). To mitigate this, we introduce a loss term based on mutual information (MI) between the original point cloud data and the learned 2D representation. Let X denote the random variable representing features of the original 3D points (this could include spatial coordinates, intensity, etc.), and Z denote the features of the corresponding 2D projection (for instance, the fused feature map or the final logits for those points). We want the 2D representation to retain as much information about the 3D data as possible. The mutual information between X and Z is defined as:

(24)

which measures how much knowing X reduces uncertainty about Z (and vice versa). High mutual information means the two representations are strongly linked (the 2D projection encodes a lot of the original 3D data). We cannot directly maximize I(X;Z) easily, but we can add a loss to discourage its decrease. We define

(25)

the negative mutual information (so minimizing L_info is equivalent to maximizing I(X;Z)). In practice, I(X;Z) might be estimated via techniques like a variational lower bound or using a mutual information neural estimator, but conceptually this term pushes the network to preserve 3D details in the 2D fused features. By including L_info, we guide RPINet to project the point cloud in a way that important information (like the presence of a small object or the exact shape of a structure) is not lost. This improves segmentation fidelity because the 2D feature map can still differentiate points that were distinct in 3D. For example, if two points are far apart in 3D but happen to project to neighboring pixels, maximizing mutual information will encourage the model to keep their features different enough to reflect the original distance (perhaps via color or slight intensity differences in the projected image), preventing confusion that could degrade segmentation.

3.5.2 Cross-entropy loss (L_CE).

This is the standard supervised segmentation loss that evaluates pixel-wise classification error. After projection and fusion, each point (pixel) i has a predicted probability distribution over the C classes (from the softmax output of the segmentation head) and a one-hot ground-truth label vector y_i (where if the true class of point i is c, otherwise 0). The cross-entropy loss for point i is . Averaged (or summed) over all points:

(26)

This term directly optimizes the per-point (pixel) classification accuracy. Minimizing L_CE encourages the model to assign high probability to the correct class for each point. It provides strong gradient signals especially when the model is very wrong (e.g., if the true class probability is near 0, the loss is large). In our multi-modal setting, cross-entropy alone ensures that each fused feature vector is pushed towards the correct class label. However, cross-entropy treats each point independently and does not account for class imbalance or spatial correlation, which is why we include the following complementary losses.

3.5.3 Dice Loss (L_Dice).

The Dice loss is based on the Sørensen-Dice coefficient, commonly used in segmentation tasks to handle class imbalance and directly maximize overlap between predicted and true regions. For a given class c, define the predicted set of points as P_c and the ground truth set as G_c. The Dice coefficient for class c is:

(27)

which ranges from 0 (no overlap) to 1 (perfect overlap). In terms of the prediction vector and one-hot label y_i, an equivalent differentiable form is:

(28)

We turn this into a loss by averaging over classes and subtracting from 1 (so higher overlap yields lower loss):

(29)

Dice loss is particularly useful when classes are imbalanced (e.g., in urban data, “building” might have far more points than “pole” or “tree”). Cross-entropy could be dominated by large classes, whereas Dice loss will force the model to not ignore small classes, since a missing small region dramatically lowers the Dice score for that class. By minimizing L_Dice, RPINet is encouraged to maximize the overlap of its predictions with ground truth for each class, improving the segmentation of both majority and minority classes. In effect, Dice loss adds a region-level focus: it cares about correctly predicting all points of an object (true positives) and not predicting too many that are not there (false positives), complementing cross-entropy’s pointwise focus.

3.5.4 IoU loss (L_IoU).

The Intersection-over-Union (IoU) metric, or Jaccard index, is another overlap-based measure that is harsher than Dice for penalizing discrepancies. For class c, . We define the IoU loss as:

(30)

IoU is similar to Dice but puts more penalty on false positives and false negatives when the regions only partially overlap. By minimizing L_IoU, the model is pushed to achieve high precision and recall for each class segmentation. In particular, this loss sharpens the segmentation boundaries and overall shapes: it is not enough to just cover the ground truth region (as Dice would emphasize), but one must also avoid covering outside the ground truth. In practice, the combination of Dice and IoU losses helps the network output segmentations that are both inclusive of all true pixels and exclusive of wrong pixels. For example, consider a thin object like a lamppost: cross-entropy might struggle if it’s a small fraction of points; Dice will ensure we try to catch it; IoU will further ensure we do not predict it too thick or in slightly the wrong place, as any extra predicted area not in ground truth hurts IoU.

3.5.5 Smoothness regularization (L_smooth).

While the above losses focus on accuracy, we also include a spatial smoothness prior to discourage isolated misclassifications and encourage coherence in the predicted segmentation map (especially in the 2D projection domain). We implement this by penalizing differences in the prediction between neighboring pixels in the 2D projection image. Define a neighborhood N_2D(i) for each pixel (for instance, the 4-connected or 8-connected neighboring pixels on the grid). We treat as a vector of class probabilities for pixel i. A smoothness loss can be written as:

(31)

where we sum the squared difference between the prediction vectors of adjacent pixels i and j. This is analogous to a discrete Laplacian or total variation regularizer on the output map, which tries to make neighboring pixels have similar label distributions. The effect is that the model is penalized if it produces a “noisy” segmentation where labels fluctuate rapidly from pixel to pixel without a strong reason. Minimizing L_smooth encourages more uniform regions of the same class, which aligns with the fact that objects or surfaces in the real world are usually composed of contiguous points. This regularization is careful not to oversmooth real boundaries: at true object edges, the cross-entropy, Dice, and IoU losses already strongly encourage the model to change class, so the smoothness loss will be counteracted by those terms. Essentially, L_smooth removes salt-and-pepper artifacts (e.g., a single point of building predicted amidst a road region) unless the data truly supports such a change. In the context of RPINet, where predictions are made on the 2D projected image, this loss helps maintain spatial consistency in that image plane which translates back to spatial consistency in 3D as well (neighboring points in the cloud that fall on nearby pixels should have the same label if they belong to the same surface).

Overall, this multi-term loss ensures that RPINet not only achieves high point-wise classification accuracy (through cross-entropy) but also yields segmentations that are information-preserving (through L_info), balanced across classes and overlapping well with ground truth regions (through Dice and IoU losses), and spatially coherent (through the smoothness term). By adjusting the weights , we can emphasize certain aspects; for example, a higher forces the model to be very faithful to 3D structure (useful if projection distortion is a concern), while higher will produce cleaner segmentation maps at the potential cost of some boundary precision. In our experiments, we find that incorporating all these terms yields the best overall performance, as each term addresses a different failure mode of large-scale segmentation: L_info protects against losing 3D details during projection, L_CE optimizes per-point accuracy, L_Dice and L_IoU improve region-level and boundary segmentation quality (especially for small or imbalanced classes), and L_smooth improves the visual and qualitative coherence of the predictions. Consequently, the trained RPINet model produces segmentation outputs that are not only accurate in labeling each point, but also preserve the integrity of objects and surfaces across the entire urban scene.