Fig 1.
The architecture of our proposed SIEANs.
The SIEANs mainly comprises of two adversarial modules: the generator and discriminator. The generator, composed of the feature learning layer (convolutional neural networks), the structural learning layer (long short-term memory networks), and the feature fusion layer (multiple convolutional layers with softmax function), performs pixel-wise scene labeling in an end-to-end fashion. Meanwhile, the discriminator (convolutional neural networks) competes with the generator via an adversarial training method.
Fig 2.
The principle of the feature learning layer.
We utilize the modified Deeplab networks to extract the convolutional features from RGB images, and select the features from the 2-th, 3-th, and 5-th layer of the Deeplab networks, then upsample the different scales of features into the same size of input images, finally concatenate them to produce the hierarchical visual features.
Fig 3.
The illustration of the structural learning layer.
The four uni-dimensional LSTMs sweep over the hierarchical visual features or the obtained hidden states pixel by pixel along four different directions, respectively: (1) from top to bottom (∨), (2) from bottom to top (∧), (3) from left to right (>), and (4) from right to left (<). And all the pixel-level hidden states learnt from the four different LSTMs are collected to produce the spatial structure features.
Fig 4.
The illustration of the feature fusion layer.
The hierarchical features and (three-dimensional) spatial structure features learnt from RGB-(D) images are concatenated to generate the multiple modal features (MMFs), then multiple convolutional layers are utilized to fuse the MMFs layer by layer, finally softmax function is exploited to perform pixel-wise scene labeling according to the fused MMFs.
Fig 5.
The architecture of the generator for RGB-D scene parsing.
In the feature learning layer, two parallel CNNs are utilized to extract the HVFs and HGFs from RGB and HHA images, respectively. In the structural learning layer, two parallel modules composed of four uni-dimensional LSTMs sweep over the HVFs and HGFs pixel by pixel along four different directions, respectively. In the feature fusion layer, multiple convolutional layers achieve the multi-modal features fusion by learning the HVFs, HGFs, and 3D-SSFs together, then softmax function is exploited to perform pixel-wise scene labeling according to the fused MMFs.
Fig 6.
The illustration of the adversarial training layer.
The discriminator takes RGB images and their corresponding ground truths (or predicted segmentations) as input, and outputs the product encodings into a convolutional neural network which consists of five stacked convolutional and max-pooling layers, a global average pooling layer, and two fully-connected layers.
Fig 7.
The relationship between the performance and computational efficiency of the SIEANs under different combinations of the HVFs on PASCAL VOC 2012 dataset.
The number of feature maps learnt from the 1-th, 2-th, 3-th, 4-th, and 5-th layer of the Deeplab networks is 64, 256, 512, 1024, and 2048, respectively.
Table 1.
The ablation study of the SIEANs on PASCAL VOC 2012 dataset without and with COCO dataset.
Table 2.
The comparison experimental results of the SIEANs with the previous state-of-the-art methods on PASCAL VOC 2012 dataset without and with COCO dataset.
We report the mean IoU on every object class.
Fig 8.
The object parsing visual results on PASCAL VOC 2012 dataset via the standard and adversarial training method.
‘SIEANs_STD’ means the visual results generated by the SIEANs via the standard training method, and ‘SIEANs’ means the visual results generated via the adversarial training method.
Table 3.
The comparison experimental results (pixel accuracy, mean accuracy, and mean IoU) of the SIEANs with different methods on SIFT FLOW dataset.
Fig 9.
The outdoor scene parsing visual results on SIFT FLOW dataset via the standard and adversarial training method.
‘SIEANs_STD’ means the visual results generated by the SIEANs via the standard training method, and ‘SIEANs’ means the visual results generated via the adversarial training method.
Table 4.
The ablation study of the SIEANs on PASCAL Person-Part dataset.
Table 5.
The comparison experimental results of the SIEANs with the previous state-of-the-art methods on PASCAL Person-Part dataset.
We report the mean IoU on every object class.
Fig 10.
The outdoor scene parsing visual results on PASCAL Person-Part dataset via the standard and adversarial training method.
‘SIEANs_STD’ means the visual results generated by the SIEANs via the standard training method, and ‘SIEANs’ means the visual results generated via the adversarial training method.
Table 6.
The comparison experimental results of the SIEANs with the previous state-of-the-art methods on Cityscapes dataset.
We report the mean IoU on every semantic class.
Table 7.
The comparison experimental results (pixel accuracy, mean accuracy, and mean IoU) of the SIEANs with different methods on Stanford Background dataset.
Fig 11.
The outdoor scene parsing visual results on Stanford Background dataset via the standard and adversarial training method.
‘SIEANs_STD’ means the visual results generated by the SIEANs via the standard training method, and ‘SIEANs’ means the visual results generated via the adversarial training method.
Fig 12.
The evolution of the mean accuracy as the number of epochs increases on the training and testing set via the standard and adversarial training, respectively.
‘SIEANs_STD’ means the SIEANs optimized via the standard training method, and ‘SIEANs’ means the SIEANs optimized via the adversarial training method.
Table 8.
The ablation study of the SIEANs on NYUDv2 dataset including or not including depth images.
Table 9.
The comparison experimental results (pixel accuracy, mean accuracy, and mean IoU) of the SIEANs with the previous state-of-the-art methods on NYUDv2 dataset including or not including depth images.
Fig 13.
The indoor scene parsing visual results on NYUDv2 dataset including or not including depth images.
‘SIEANs_RGB’ means the visual results generated by the SIEANs on RGB images, and ‘SIEANs_RGB-D’ means the visual results generated by the SIEANs on RGB-D images.
Table 10.
The comparison experimental results (pixel accuracy, mean accuracy, and mean IoU) of the SIEANs with the previous state-of-the-art methods on SUN-RGBD dataset including or not including depth images.