Visual design elements.

Visual design elements refer to the fundamental components that constitute a visual design, including shapes, colors, lines, textures, and space. These elements are combined and organized to convey specific visual information, emotions, and themes, playing a critical role in both artistic and functional design. They serve as the building blocks through which designers can express creativity, structure, and intent. The key types of visual design elements include shapes and graphics, which influence the form and visual appeal; colors, which communicate emotions and establish atmosphere; lines, which define space and rhythm; textures, which enhance depth and tactile perception; and space, which determines the organization and clarity of the design. [37–39]

The use of visual design elements holds significant value in various design fields, including branding, user interface design, advertising, and product packaging. These elements not only enhance the aesthetic appeal and recognizability of a design but also guide the user’s visual experience, contributing to greater engagement and brand loyalty. By carefully combining and innovating with these elements, designers can achieve more precise and effective communication, increasing the competitive edge of their designs in the market. Furthermore, the strategic application of these elements allows for the creation of designs that are not only visually striking but also functional, ensuring that they resonate with users on both emotional and practical levels.

Overview of visual design element recognition techniques.

Visual design element recognition [40] refers to the automated process of identifying and classifying fundamental components of visual designs, such as shapes, colors, lines, textures, and spatial arrangements [41]. This process is critical in various fields, including digital design analysis, content-based image retrieval [42], art and fashion recognition, and design optimization. By enabling machines to understand and interpret the building blocks of visual compositions, these techniques facilitate more intelligent design analysis, pattern recognition, and automation in creative workflows [43]. The underlying principle of visual design element recognition involves extracting and analyzing low- to high-level features from visual data and mapping these features to predefined categories or design elements.

The techniques used for visual design element recognition can be broadly classified into several categories, including traditional image processing, feature-based methods, deep learning-based methods, and hybrid approaches that combine multiple techniques. Each of these methods has its own set of strengths and weaknesses depending on the complexity of the design environment and the specific requirements of the task.

• Traditional Image Processing Methods: These methods rely on low-level image processing techniques, such as edge detection [44], thresholding [45], color histograms [46], and texture segmentation [47]. Common algorithms include Canny edge detection [48], Sobel filters [49], and Fourier transforms [50]. These methods are computationally efficient and simple to implement, making them suitable for real-time applications and resource-constrained environments. However, they struggle with complex, non-linear relationships in design elements, and are less effective when designs contain noise, occlusions, or intricate textures. Additionally, traditional methods often fail to capture the semantic meaning of design elements, limiting their effectiveness in tasks requiring deeper contextual understanding. [51]
• Feature-Based Recognition Methods: These approaches focus on detecting distinctive features from images, such as corners, edges, or keypoints. Algorithms like SIFT (Scale-Invariant Feature Transform) [52], HOG (Histogram of Oriented Gradients) [53], and SURF (Speeded Up Robust Features) [54] extract these features to match and classify design elements. Feature-based methods offer better performance than traditional methods, especially in handling variations in scale, rotation, and lighting. They are widely used for object recognition and scene parsing. However, these methods tend to be computationally expensive, and their performance can degrade in the presence of cluttered or noisy design elements. Moreover, they typically focus on specific elements like shapes and textures, without capturing higher-level design structures or semantic meaning. [55]
• Deep Learning-Based Methods: In recent years, deep learning, particularly Convolutional Neural Networks (CNNs) [56] and Transformer-based [57] models, has become the dominant approach for visual element recognition. CNNs excel at learning hierarchical features directly from raw pixel data, enabling them to identify complex patterns and relationships in visual designs. Recent innovations, such as Swin Transformer [36] and Vision Transformers (ViTs) [58], have extended these capabilities by capturing both local and global context in design elements. These methods have shown significant improvements in accuracy and generalization, especially in large-scale and complex design datasets. However, deep learning methods require large-annotated datasets for training, extensive computational resources, and can be slow to adapt to new design trends without further fine-tuning or transfer learning. Furthermore, the “black-box” nature of deep models can make it difficult to interpret the reasoning behind their classifications. [59]
• Hybrid Approaches: Hybrid methods combine traditional image processing or feature-based techniques with deep learning models to leverage the strengths of both approaches. For example, a hybrid model might first use edge detection to extract basic shapes and then apply a CNN or Transformer model to classify and interpret these shapes within the context of the design. Hybrid methods aim to improve both efficiency and robustness by providing a balance between interpretability and recognition performance. However, these methods introduce additional complexity and may increase computational costs, particularly when combining multiple deep learning models or integrating various feature extraction techniques. [60]

Recent research has made significant strides in improving the accuracy and adaptability of visual design element recognition. One of the key developments has been the integration of multi-modal models that combine visual data with textual information, often using multi-modal transformers or vision-language models such as CLIP [61]. These models can understand both the visual characteristics and the semantic meaning of design elements, leading to better recognition of context-specific design components.

Another major advancement is the use of self-supervised learning techniques, which allow models to learn from unannotated data by leveraging data augmentation, contrastive learning, or pretext tasks [62]. This approach is particularly beneficial in design contexts where labeled data is scarce or expensive to obtain. Furthermore, advancements in few-shot learning [63] and transfer learning [64] have enabled deep learning models to generalize better with smaller datasets, making it easier to apply these techniques to specific design domains or emerging design trends. In addition, spatial reasoning and attention mechanisms have become a focal point of research, particularly with the introduction of Transformer-based models. [65] These models excel at capturing long-range dependencies and complex spatial relationships in design elements, enabling more accurate recognition in intricate or dynamic design environments. Despite these advancements, challenges remain, such as handling real-time recognition in large-scale applications, improving interpretability for design professionals, and reducing the computational burden of deep learning models. Ongoing research continues to focus on enhancing model efficiency, robustness, and adaptability to new design contexts.

The combination of Swin Transformer-based segmentation with multi-scale feature extraction has been explored in previous work (e.g., semantic segmentation and fine-grained categorization tasks), while multimodal LLMs have become standard in visual language tasks (e.g., variants of CLIP or VL-BERT)(see Table 2). This makes the research likely to be seen as incremental rather than a seminal contribution. However, the target paper provides incremental improvements in specific applications of visual design elements (e.g., brand logos, packaging), especially by end-to-end integration of the three to deal with complex design layouts and semantic intents.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Comparative analysis of cross-modal model design.

https://doi.org/10.1371/journal.pone.0339277.t002

While the individual components of our framework have foundations in existing literature, the novelty of this work lies in the end-to-end integration and its specific architectural design tailored for fine-grained visual design element analysis. As detailed in Table 2, our approach differs from prior works in several key aspects. Unlike the base Swin Transformer which focuses on general hierarchical vision tasks, our framework incorporates a dedicated multi-scale feature fusion mechanism followed by a multimodal LLM fine-tuned for design semantics. This allows for a cohesive flow from precise segmentation to semantic interpretation of design intent, a progression not present in the original architecture. Compared to multi-scale fusion methods like MSCPN [67] that rely on covariance pooling, our integration with the Swin backbone provides a more powerful hierarchical feature extractor. Furthermore, while multimodal LLMs are indeed becoming standard, our application involves a specialized cross-modal attention layer post-segmentation and a domain-specific fine-tuning strategy using design-centric datasets, moving beyond the general alignment goals of models surveyed in Literature [14].

Fine-grained understanding of visual design elements based on multi-modal LLM.

Fine-grained understanding of visual design elements using multi-modal large language models (LLMs) [70] integrates both visual and textual data to enable detailed recognition and interpretation of design components, such as shapes, colors, textures, and their contextual meanings. Unlike traditional image recognition, this approach incorporates semantic information from textual descriptions, allowing for more nuanced analysis of design elements and their relationships within a composition. The goal is to capture subtle features and understand the design intent behind each element [71,72].

Multi-modal LLMs leverage both visual and textual information. Vision models, such as Convolutional Neural Networks (CNNs) or Vision Transformers (ViTs), extract detailed visual features from images, while natural language models process textual data. These models are typically fused at various stages to combine insights from both modalities. Vision-language pretraining models like CLIP and ALIGN [73]align image and text features in a shared embedding space, allowing the model to recognize design elements in fine detail by relating visual patterns to textual descriptions. Transformer-based models (e.g., VL-BERT [74]) use attention mechanisms to enhance this process by focusing on specific visual regions and their corresponding textual meanings.

The primary advantage of multi-modal LLMs is their ability to combine visual recognition with semantic context, allowing for more accurate and contextual aware understanding of design elements. This is especially useful for tasks like art interpretation, design composition analysis, and fashion recognition, where understanding the relationships between elements is key. However, these models face several challenges, including high computational costs, especially for high-resolution images or large datasets. Additionally, achieving effective fusion between visual and textual information can be difficult, leading to misalignments or inefficiencies.

Recent research has focused on improving fusion techniques, such as dynamic fusion [75–77], to better integrate visual and textual data. Self-supervised learning [78] and few-shot learning [79] are being used to enhance fine-grained recognition with minimal labeled data, addressing the challenge of domain-specific adaptation. Moreover, advancements in spatial reasoning and attention mechanisms allow models to better capture intricate design details and their contextual significance [80]. Efforts to improve model interpretability, such as attention visualization [81] and explainable AI (XAI) [82], are also advancing, making it easier to understand model decisions.

Experimental design.

Experimental Design In order to evaluate the efficacy of the multi-modal LLM-based fine-grained image understanding approach for visual design elements, several experiments were conducted. These experiments aimed to assess the model’s performance on tasks such as counting, position detection, color recognition, text recognition, celebrity identification, scene recognition, landmark detection, and artwork identification. Two main experimental categories were designed: baseline experiments, which focused on evaluating single-task performance, and integrated experiments, which tested the model’s capability in handling multiple tasks simultaneously. The experiments were conducted using well-established datasets, including COCO, ETHZ Shape Classes, and ImageNet, ensuring a comprehensive evaluation across diverse design layouts and task scenarios.

Experimental Setup The experiments were carried out on a high-performance computing workstation equipped with the following hardware and software configuration: two NVIDIA A100 40GB GPUs, 64GB of RAM, and an Intel Xeon 64-core CPU. The software stack included Python 3.8, PyTorch 1.10, CUDA 11.1, and various libraries for data preprocessing and model training, such as Transformers 4.14 and OpenCV 4.5. The operating system was Ubuntu 20.04. Data loading and preprocessing were optimized using multi-threading to accelerate the training process. Each experiment was repeated five times to ensure result stability, and the final performance metrics were averaged over these runs.

Model Configuration Key hyperparameters of the model were set as follows: The visual encoder was based on the Swin Transformer, with a hidden dimension of 384, a patch size of 4x4, 12 layers, and 12 attention heads, resulting in an output dimension of 768. The textual encoder utilized a BERT-base model with a hidden size of 768 and a maximum sequence length of 128 tokens. The cross-modal attention mechanism used a feature dimension of 512, with the query, key, and value projection matrices set to 512. The Adam optimizer was used with an initial learning rate of 1e-4 and weight decay of 1e-5, applying a cosine annealing learning rate scheduler. The batch size was set to 32, and gradient accumulation was utilized to optimize training efficiency.

Experimental Details The experimental procedure involved two main phases: pre-training and fine-tuning. During the pre-training phase, the model was initially trained on large-scale datasets such as ImageNet and COCO for 50 epochs to learn basic visual features. In the fine-tuning phase, the model was further trained on task-specific datasets for an additional 20 epochs. Data augmentation techniques, such as random cropping, rotation, and color jittering, were applied to enhance the model’s robustness. The training and evaluation datasets were split into 80% for training and 20% for validation. Additionally, 10-fold cross-validation was used to ensure the reliability and generalizability of the results.

Evaluation Metrics The performance of the model was evaluated using a variety of metrics tailored to each task. For the counting task, the mean absolute error (MAE) was computed, which quantifies the difference between predicted and ground truth counts. Position detection was evaluated using the average localization error (ALE), which measures the Euclidean distance between predicted and true bounding boxes. For color recognition, accuracy was calculated, representing the proportion of correctly classified colors. Text recognition performance was assessed using Character Error Rate (CER) and Word Error Rate (WER). In tasks such as celebrity identification, scene recognition, landmark detection, and artwork classification, the top-1 accuracy was used to evaluate the model’s classification performance. These metrics allowed for a comprehensive assessment of the model’s performance across various visual design tasks, ensuring its suitability for real-world applications.

Training Details The training process employed the Adam optimizer with an initial learning rate of and weight decay of , utilizing a cosine annealing learning rate scheduler. Training was conducted for a total of 70 epochs, comprising 50 epochs of pre-training on large-scale datasets (ImageNet and COCO) followed by 20 epochs of task-specific fine-tuning. The stopping criterion was based on monitoring the validation loss, with early stopping implemented if no improvement was observed for 10 consecutive epochs to prevent overfitting. The minimization of the combined segmentation and classification loss functions was considered achieved when the validation loss plateaued, indicating model convergence.

Model Training Details To mitigate overfitting and ensure robust generalization, several regularization techniques were employed during the training process. The loss for both the training and validation sets was meticulously monitored. Specifically, weight decay with a coefficient of was applied to all learnable parameters to constrain model complexity. Additionally, dropout layers with a rate of 0.1 were incorporated within the multi-scale feature fusion modules. Data augmentation techniques, including random cropping, rotation (), and color jittering (brightness = 0.2, contrast = 0.2, saturation = 0.2, hue = 0.1), were extensively used to increase the diversity of the training data. Early stopping was implemented based on the validation loss, halting training if no improvement was observed for 10 consecutive epochs. The comparison of training and validation loss curves confirmed the effectiveness of these regularization measures in preventing overfitting and promoting stable convergence.

Dataset and data preprocessing.

The following three datasets were used in this study.

ETHZ Shape Classes The ETHZ Shape Classes dataset is a specialized dataset designed for shape-based object recognition tasks. It contains 255 images divided into five object classes: Applelogos, Bottles, Giraffes, Mugs, and Swans. The dataset focuses on objects with strong, distinct shapes in cluttered backgrounds, challenging models to accurately identify contours and boundaries. Images are provided in varying resolutions, ensuring diverse and realistic scenarios. Key fields in the dataset include object masks, object class labels, and image metadata. In this study, ETHZ Shape Classes is particularly advantageous for evaluating shape recognition capabilities, as its emphasis on distinctive object contours aligns well with the model’s segmentation and recognition tasks.

ImageNet The ImageNet dataset is a large-scale visual database widely used for pre-training deep learning models. It comprises over 14 million labeled images spanning 1,000 object classes, making it a cornerstone for developing robust image recognition systems. Images in ImageNet are high-resolution and come with class labels and associated bounding boxes for object localization tasks. The dataset’s diversity and scale make it ideal for pre-training the Swin Transformer in this study, allowing the model to learn generalizable visual features. Its comprehensive coverage of object categories also facilitates the development of recognition models that perform well on diverse and complex design elements.

COCO The COCO (Common Objects in Context) dataset is a comprehensive dataset for object detection, segmentation, and captioning tasks. It contains over 330,000 images, including 200,000 labeled with 80 object categories and 1.5 million object instances. Key fields include object bounding boxes, segmentation masks, image captions, and object categories. COCO stands out for its emphasis on contextual relationships between objects, providing a rich resource for models requiring both fine-grained object detection and contextual understanding. In this study, COCO is used to evaluate the model’s ability to segment and analyze visual design elements in complex, multi-object scenes, leveraging its annotated masks and captions for fine-grained multimodal tasks.

For data preprocessing, we implemented a rigorous pipeline to maintain consistency while preserving design integrity. This included standardized normalization procedures to handle variations in image formats and resolutions. Spatial transformations were applied judiciously to augment training data without distorting essential design characteristics. All preprocessing steps were validated through visual inspection to ensure they maintained the semantic meaning of design elements.

The augmentation strategy was specifically designed to address common challenges in design recognition tasks. We employed geometric transformations that respect design principles, such as aspect-ratio-preserving scaling and rotation within reasonable bounds. Color space adjustments were carefully calibrated to maintain the perceptual qualities of design elements while introducing necessary variability.

Sample Dataset Implementation.

To further elucidate the proposed framework, we provide a brief implementation on a sample dataset. The sample comprises two representative images: one containing a logo design and another with a textured pattern. The image is first processed by the Swin Transformer for segmentation. For instance, the logo image is segmented into distinct regions corresponding to shapes and text. The segmented output is then passed through the multi-scale feature fusion module. Features extracted at different scales are integrated, enhancing the recognition of element types such as ‘circle’ for the logo shape and ‘serif font’ for the text. Finally, the multimodal LLM analyzes the fused visual features alongside a textual prompt. The LLM’s cross-modal attention mechanism aligns the visual segments with semantic concepts, outputting a fine-grained understanding like “A circular logo with bold, serif text conveying a classic brand identity.” This step-by-step demonstration on sample data clarifies the integrated workflow of our framework.

Comparison study with SOTA models.

To comprehensively evaluate the performance of the proposed model, a series of comparative experiments were conducted against several state-of-the-art (SOTA) models, including DETR, Mask R-CNN, Vision Transformer (ViT), and LayoutLMv3. The experiments focused on three key datasets: ETHZ Shape Classes, ImageNet, and COCO. The evaluation metrics included segmentation accuracy, mean Average Precision (mAP) for object detection, and F1 score for multimodal tasks such as text-guided element recognition. Each model was trained and tested under identical experimental conditions to ensure fair comparison. The results, summarized in Table 2, demonstrate the superiority of the proposed model in addressing various visual design tasks.

Detailed Results On the COCO dataset, the proposed model achieved a segmentation accuracy of 88.6%, significantly outperforming DETR (83.4%), Mask R-CNN (85.2%), and ViT (84.8%). For object detection tasks, the proposed model obtained a 51.4% mAP, compared to 47.2% for DETR, 49.3% for Mask R-CNN, and 48.7% for ViT(see Fig 5). In multimodal tasks, such as text-guided element recognition, the proposed model achieved an F1 score of 92.3%, which is markedly higher than LayoutLMv3’s 88.5%(see Table 3).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Comparison of the Proposed Model with SOTA Models.

https://doi.org/10.1371/journal.pone.0339277.t003

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. Performance comparison with SOTA Models.

https://doi.org/10.1371/journal.pone.0339277.g005

On the ETHZ Shape Classes dataset, which emphasizes shape recognition in cluttered scenes, the proposed model demonstrated outstanding performance with a classification accuracy of 93.1%, surpassing Mask R-CNN (89.7%) and ViT (91.2%). This highlights the proposed model’s effectiveness in capturing fine-grained visual features and distinguishing complex shapes.

On the ImageNet dataset, the proposed model achieved a top-1 accuracy of 92.4%, outperforming ViT (90.1%) and other SOTA models. The results confirm the proposed model’s ability to generalize across large-scale datasets and handle diverse visual categories effectively.

The proposed model’s multi-layered approach, combining Swin Transformer-based segmentation, multi-scale feature fusion for recognition, and multimodal LLM for fine-grained understanding, provides substantial improvements across all tasks. The high segmentation accuracy and object detection mAP on COCO demonstrate its ability to handle complex scenes with multiple objects. The high F1 score for text-guided tasks highlights its strength in multimodal reasoning, which is essential for understanding contextual and semantic relationships in visual design. These results establish the proposed model as a robust and versatile tool for visual design analysis, outperforming SOTA models on diverse benchmarks.

Comparison study.

Five comparative experiments were then conducted to verify the performance of the proposed model from different perspectives.

Comparison Study 1 Segmentation Accuracy on Complex Shapes. This experiment compared the segmentation accuracy of the proposed model against DETR, Mask R-CNN, and ViT on the ETHZ Shape Classes dataset. The dataset’s focus on objects with complex shapes and cluttered backgrounds made it an ideal benchmark for evaluating shape segmentation performance. Models were tested on 255 images across five shape classes, with segmentation accuracy measured using the Intersection over Union (IoU) metric. The proposed model achieved an IoU of 87.5%, significantly outperforming DETR (80.2%), Mask R-CNN (84.1%), and ViT (82.3%)(see Table 4).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 4. Experimental results table of comparison study 1.

https://doi.org/10.1371/journal.pone.0339277.t004

The proposed model’s superior performance is attributed to its Swin Transformer-based segmentation, which captures both local and global shape features effectively(see Fig 6). The multi-scale attention mechanism ensures fine-grained segmentation even in cluttered backgrounds, while DETR and Mask R-CNN struggle to maintain accuracy in such complex scenarios due to their reliance on pre-defined anchors or less robust global context modeling.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 6. Segmentation Accuracy on ETHZ Shape Classe.

https://doi.org/10.1371/journal.pone.0339277.g006

Comparison Study 2 Object Detection in Multi-Object Scenes. This study compared the object detection performance of the proposed model with DETR, YOLOv5, and Mask R-CNN on the COCO dataset. The test set included images containing multiple overlapping objects, and the detection performance was evaluated using mean Average Precision (mAP) at an IoU threshold of 0.5. The proposed model achieved mAP of 51.4%, outperforming DETR (47.2%), YOLOv5 (48.6%), and Mask R-CNN (49.3%)(see Table 5).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 5. Experimental results table of comparison study 2.

https://doi.org/10.1371/journal.pone.0339277.t005

The advantage of the proposed model lies in its multi-scale feature fusion, which effectively integrates low-level spatial features and high-level semantic information(see Fig 7). This enables better detection of small and occluded objects, whereas DETR and YOLOv5 struggle with occlusion, and Mask R-CNN exhibits limited performance due to reliance on fixed-region proposals.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 7. COCO dataset performance comparison.

https://doi.org/10.1371/journal.pone.0339277.g007

Comparison Study 3 Multimodal Text-Guided Recognition. The experiment evaluated the text-guided recognition performance of the proposed model and LayoutLMv3 on the COCO dataset. Tasks included recognizing visual elements based on textual descriptions, such as “red circle” or “landmark Eiffel Tower.” The F1 score was used as the evaluation metric. The proposed model achieved an F1 score of 92.3%, outperforming LayoutLMv3, which scored 88.5%(see Table 6).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 6. Experimental results table of comparison study 3.

https://doi.org/10.1371/journal.pone.0339277.t006

The superiority of the proposed model is due to its multimodal large language model (LLM), which effectively combines textual and visual features(see Fig 8). The cross-modal attention mechanism enhances the understanding of semantic relationships between textual descriptions and visual elements, while LayoutLMv3 struggles with complex text-visual correlations in detailed scenarios.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 8. Multimodal text-guided recognition F1 scores.

https://doi.org/10.1371/journal.pone.0339277.g008

Comparison Study 4. Color and Texture Recognition. This study compared the proposed model with Mask R-CNN, ViT, and DETR for color and texture recognition tasks on a subset of the ImageNet dataset. Accuracy in correctly identifying the dominant color and texture was used as the evaluation metric. The proposed model achieved a recognition accuracy of 94.2%, surpassing Mask R-CNN (89.8%), ViT (91.4%), and DETR (88.7%)(see Table 7).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 7. Experimental results table of comparison study 4.

https://doi.org/10.1371/journal.pone.0339277.t007

The proposed model’s higher accuracy is attributed to its multi-scale feature fusion, which combines spatial and textural information(see Fig 9). This allows for more precise recognition of subtle textures and color variations, while other models are limited by less sophisticated feature aggregation techniques.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 9. Color and texture recognition performance.

https://doi.org/10.1371/journal.pone.0339277.g009

Comparison Study 5 Scene and Landmark Recognition. This experiment evaluated the scene and landmark recognition capabilities of the proposed model against ViT, LayoutLMv3, and YOLOv5 on the COCO dataset. The task involved identifying scenes (e.g., beach, forest) and landmarks (e.g., Eiffel Tower) with top-1 accuracy as the evaluation metric. The proposed model achieved a top-1 accuracy of 93.7%, compared to ViT (90.3%), LayoutLMv3 (88.9%), and YOLOv5 (89.6%)(see Table 8).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 8. Experimental results table of comparison study 5.

https://doi.org/10.1371/journal.pone.0339277.t008

The proposed model excels in scene and landmark recognition due to its fine-grained image understanding enabled by the multimodal LLM(see Fig 10). The integration of visual and contextual information allows it to discern subtle cues in scene composition and landmark features. Other models lack the advanced multimodal attention mechanisms required for such nuanced understanding.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 10. Scene and landmark recognition performance.

https://doi.org/10.1371/journal.pone.0339277.g010

Ablation study results and analysis.

To validate the contributions of each module in the proposed model, we conducted a series of ablation experiments(results are shown in Fig 11). The experiments evaluated the impact of removing or replacing key components: Swin Transformer for segmentation, multi-scale feature fusion, and the multimodal large language model (LLM) for fine-grained image understanding. Performance was measured using segmentation accuracy, mAP for object detection, and F1 score for multimodal tasks across the COCO dataset.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 11. Ablation study: component contribution analysis.

https://doi.org/10.1371/journal.pone.0339277.g011

Ablation study #1: Impact of Swin Transformer for Segmentation

We replaced the Swin Transformer module with a standard ResNet-50-based segmentation backbone to assess its contribution. Segmentation accuracy and IoU were measured for visual design element segmentation on the COCO dataset.

The Swin Transformer significantly improved segmentation accuracy (+4.3%) and IoU (+6.3%)(see Table 9). This is due to its ability to capture hierarchical global and local features through shifted window attention, which is particularly effective in complex layouts. ResNet-50 lacks this nuanced hierarchical feature aggregation, leading to poorer performance.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 9. Experimental results table of ablation study #1.

https://doi.org/10.1371/journal.pone.0339277.t009

Ablation study #2: Impact of Multi-Scale Feature Fusion

The multi-scale feature fusion module was removed, and a single-scale feature extraction approach was employed. Object detection mAP was measured to evaluate performance on the COCO dataset.

The multi-scale feature fusion improved object detection mAP by 4.7%(see Table 10). This module enhances performance by combining low-level spatial and high-level semantic features, enabling the model to detect objects of varying sizes and complexities. Without multi-scale fusion, the model struggled with smaller objects and occluded scenarios.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 10. Experimental results table of ablation study #2.

https://doi.org/10.1371/journal.pone.0339277.t010

Ablation study #3: Impact of Multimodal LLM for Fine-Grained Understanding

The multimodal LLM was replaced with a traditional image-only CNN for analyzing text-guided tasks, such as color recognition and landmark identification. F1 scores were measured for text-guided recognition on COCO.

The multimodal LLM significantly improved the F1 score (+5.8%)(see Table 11). Its cross-modal attention mechanism effectively integrates textual and visual information, enabling nuanced understanding of context and semantics. The image-only CNN lacked the capacity to leverage textual guidance, resulting in lower performance.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 11. Experimental results table of ablation study #3.

https://doi.org/10.1371/journal.pone.0339277.t011

The ablation experiments validate the effectiveness of the proposed model’s modular design. Each component provides critical improvements to specific tasks, and their combined use ensures robust, high-performance visual design element analysis. These findings highlight the innovative and complementary nature of the proposed architecture.

Ablation Study #4: Impact of Integrated Pipeline Architecture

To rigorously validate the synergistic contribution of the proposed integrated pipeline, which combines Swin Transformer-based segmentation, multi-scale feature fusion, and multimodal LLM for fine-grained design element recognition, we conducted an ablation study to assess the impact of removing individual components. The experiment evaluates the full model against variants where: (1) the Swin Transformer is replaced with a ResNet-50 backbone (lacking hierarchical attention), (2) multi-scale feature fusion is removed in favor of single-scale features, and (3) the multimodal LLM is substituted with a CNN-based image-only model. Performance was measured across the ETHZ Shape Classes dataset (segmentation, IoU), ImageNet dataset (color/texture recognition, accuracy), and COCO dataset (object detection, mAP; text-guided recognition, F1 score). The results, presented in Table 1, demonstrate significant performance degradation when any component is removed, confirming that the integrated architecture drives the model’s superior performance (e.g., 92.3% F1 score on COCO) through the synergistic combination of components, rather than the isolated strengths of Swin Transformer’s efficiency or LLM’s contextual understanding.

The results in Table 12 highlight the critical role of each component. Removing the Swin Transformer reduces IoU by 6.3% on ETHZ Shape Classes due to the loss of hierarchical shifted window attention, which is essential for capturing complex design layouts. Omitting multi-scale feature fusion leads to a 4.7% drop in COCO mAP, as single-scale features struggle with objects of varying sizes. Replacing the multimodal LLM with a CNN-based model causes a 5.8% decrease in COCO F1 score, underscoring the LLM’s importance in aligning visual and textual semantics for fine-grained design intent interpretation. These findings confirm that the proposed pipeline’s novelty lies in its integrated architecture, which synergistically combines these components to achieve robust performance across diverse design recognition tasks.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 12. Experimental Results of Ablation Study #4: Integrated Pipeline Contribution.

https://doi.org/10.1371/journal.pone.0339277.t012

Ablation Study on Integrated Framework Contribution

To quantitatively validate that the performance superiority stems from the synergistic integration of the proposed framework’s components rather than their isolated capabilities, we conducted a comprehensive ablation study. The full model was compared against three variants: 1) w/o Swin: The Swin Transformer was replaced with a ResNet-50 backbone for segmentation; 2) w/o Multi-Scale: The multi-scale feature fusion module was removed, using only single-scale features from the final layer of the backbone; 3) w/o LLM: The multimodal LLM was replaced with a standard CNN-based model for image-only understanding, disabling cross-modal analysis. Each variant was evaluated on the ETHZ Shape Classes (segmentation IoU), ImageNet (color/texture recognition accuracy), COCO (object detection mAP), and COCO (text-guided recognition F1 score) datasets. The results demonstrate a significant and consistent performance drop across all tasks and datasets when any key component is ablated. For instance, on the COCO dataset, removing the multimodal LLM led to a 5.8% decrease in the F1-score for text-guided recognition, while replacing the Swin Transformer caused a 4.6% reduction in object detection mAP. These results conclusively show that the high performance reported is not attributable to any single component but is a direct outcome of their carefully engineered integration within the end-to-end pipeline, thereby solidifying the framework’s novelty.