https://orcid.org/0000-0001-9422-0378
Figures
Abstract
Designers rely on sketching to visualize and refine their initial ideas, and virtual reality (VR) tools now facilitate sketching in immersive 3D environments. However, little research has been conducted on the differences in the visual and spatial processes involved in 3D versus 2D sketching and their effects on cognition. This study investigated potential differences in spatial and visual functions related to the use of 3D versus 2D sketching media by analyzing functional magnetic resonance imaging (fMRI) data. We recruited 20 healthy, right-handed students from the Department of Horticulture and Landscape Architecture with at least three years of experience in freehand landscape drawing. Using an Oculus Quest VR headset controller and a 12.9-inch iPad Pro with an Apple Pencil, we tested participants individually with 3D and 2D sketching, respectively. When comparing 2D and 3D sketches, our fMRI results revealed significant differences in the activation of several brain regions, including the right middle temporal gyrus, both sides of the parietal lobe, and the left middle occipital gyrus. We also compared different sketching conditions, such as lines, geometrical objects (cube), and naturalistic objects (perspective view of a tree), and found significant differences in the spatial and visual recognition of brain areas that support visual recognition, composition, and spatial perception. This finding suggests that 3D sketching environments, such as VR, may activate more visual–spatial functions during sketching compared to 2D environments. The result highlights the potential of immersive sketching environments for design-related processes and spatial thinking.
Citation: Tung Y-H, Chang C-Y (2024) How three-dimensional sketching environments affect spatial thinking: A functional magnetic resonance imaging study of virtual reality. PLoS ONE 19(3): e0294451. https://doi.org/10.1371/journal.pone.0294451
Editor: Xiong Jiang, Georgetown University Medical Center, UNITED STATES
Received: January 19, 2023; Accepted: October 31, 2023; Published: March 11, 2024
Copyright: © 2024 Tung, Chang. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: The data set necessary to replicate our study's findings has been uploaded to a stable, public repository - Harvard Dataverse. The relevant information is as follows: Title: How three-dimensional sketching environments affect spatial thinking: A functional magnetic resonance imaging study of virtual reality DOI: https://doi.org/10.7910/DVN/NXEKBR Repository: Harvard Dataverse Version: V1.
Funding: Chun-Yen Chang received the research funding from the National Science and Technology Council, Taiwan. Grant No. 106-2410-H-002-173-MY3. https://www.nstc.gov.tw/ The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: The authors have declared that no competing interests exist.
Introduction
Sketching is an important part of the design thinking process, as it allows designers to visualize and experiment with their ideas. As Aristotle once wrote, “the soul never thinks without a mental image” (p. 177) [1]. He illustrates how the visualization of an image is critical to thinking. For landscape designers, sketching is an essential way to represent an image conceived in the mind, which is a critical means of constructing arguments for the design task [2]. Visualization through sketching helps to improve the creation and manipulation of space and form and conveys the conversation of imagination [3]. Through sketching, designers envisage better solutions and possibilities with more precise solution than with creative thinking alone [4, 5]. As technology has developed, many tools now aid designers in visualizing their sketches. In terms of sketching media, the tools can be broadly classified as two-dimensional (2D) tools (e.g., pencil-and-paper and graphics tablets) and three-dimensional (3D) tools, such as computer modeling and VR (e.g., computer-aided design systems and gesture-based interfaces) [6]. Wu et al. [7] found that digital media has made it possible for designers to produce in-depth descriptions of detailed designs with the help of related functions in digital media, while the traditional pen-and-paper environment allows designers to use synthetic and contrasting methods to manipulate their idea development.
As a type of sketching media, VR has recently become increasingly popular in many research fields, including environmental simulation, technical operations, and design education [8–10]. VR refers to a computer-generated experience that creates a sense of immersion and presence [11]. It is a powerful tool for increasing design concept generation [12] and operation training performance [13], and it is widely used in design review visualization media [14]. The efficiency of VR sketching media for practicing spatial ability has prompted their increased use in environmental simulations, technical operations, and design-related educational practices [8–10]. According to the degree of immersion, VR systems can be assigned to three categories: nonimmersive (desktop display), semi-immersive (screen-projected display), and immersive (head-mounted gesture display) [15]. Gesture-based immersive VR interfaces permit 3D environments to create the full virtual sensation of interacting on a real-world scale through headsets and remote controllers [16, 17]. A recent study by [18] found that immersive VR head-mounted displays are more suitable than 2D screens for training 3D movements because they are more motivating to user and easy to operate. Paes et al. [19] also found that immersive VR systems provide users with an enhanced 3D perception of virtual models and greater levels of presence compared to nonimmersive systems, which can benefit collaborative design reviews and increase productivity. These articles suggest that immersive VR tools can enhance users’ perceptions and understanding of a 3D space while consuming fewer cognitive resources during mental imaging (e.g., molecular structure, materials, and components) [15–17, 20–22]. Furthermore, VR tool is considered to provide higher experimental control over stimuli, with validity as an investigational medium for capturing brain activation profiles [23]. This experience of immersion, interaction, and involvement supports enhancing the perception and understanding of the 3D space [24, 25].
Research by [26] found that visualization environments that better match human perceptual and interaction capabilities to the task at hand improve our understanding of 3D visualizations. Similarly, [27] showed that using an immersive virtual environment in a design studio can enhance spatial perception, improve designs, and affect the design process by changing an individual’s method of thinking in architectural design. Therefore, 3D sketching tools, such as VR, have great potential for practicing spatial and design thinking in the realm of spatial visualization [9, 10]. A distinguished example is “Foldit” [28], an online game that allows players to design stable protein structures to advance their understanding of protein folding and prediction. 3D visualization is essential to making this process more accessible compared to traditional methods, which are extremely difficult and time-consuming.
However, most studies in design-related fields have primarily focused on comparing the outcomes of practices in 2D- and 3D-based sketching environments, a vital question remained unanswered in the brain activation caused by using different sketching tools. In empirical research, the mechanism through which viewing sketches in a 3D virtual environment affects the brain has been less studied. The current study investigated the differences in brain activation observed when participants viewed 2D and 3D environmental sketches. The study did not consider movement or the ability to sketch but instead focused on the spatial processes involved in viewing different sketching tools and sketching tasks that affect the visual and spatial functions of the brain. This research aims to investigate the neural activation involved in viewing the spatial sketching process and to distinguish the effects of brain activation caused by 2D and 3D visual sketching environments.
The role of spatial processing in sketching and its neural correlates
This section discusses the importance of spatial representation in sketching and highlights the differences between 2D and 3D sketching environments. Sketching involves precise perceptual processing, especially in a 3D spatial context in which locations and objects play a role in spatial representation. Viewing a sketch involves a complex set of brain processes connected to perceiving, processing, and interpreting visual information. Sketching with a 3D virtual tool permits users to create higher-fidelity visualizations of spatial depth and space navigation than sketching with a 2D tool [29]. When sketching using a 2D tool, our perception transforms and manipulates the image into a 3D image; the representation of these characteristics depends not only on the objects’ arrangement, orientation, and perspective scale but also on the information conveyed by 2D and 3D spatial transformation views. In contrast, when sketching with a 3D tool, the visualization of the depth and navigation provided by the computer helps the user visualize the object as a whole [30–32].
Spatial processing refers to the cognitive functions of perceiving, analyzing, and manipulating information about the spatial relationships between objects and locations in the environment; these processes are symmetrically distributed in both hemispheres [33]. During spatial processing, the brain achieves spatial representation through two distinct neural pathways: the dorsal and ventral pathways. The dorsal pathway, also known as the “where” pathway, is responsible for processing information related to the location and movement of objects in the environment. This pathway is involved in forming egocentric representations of spatial context, spatial navigation, and motor planning. In contrast, the ventral pathway, also known as the “what” pathway, is responsible for processing information related to the identity and properties of objects in the environment. This pathway is involved in forming allocentric representations of spatial context, object recognition, and categorization. Both dorsal and ventral pathways are interconnected and work together to facilitate a complete understanding of visual information [33–36]. The dorsal pathway is primarily associated with the posterior parietal cortex and the medial temporal lobe, while the ventral pathway is associated with the inferior temporal cortex and the occipital lobe [33–35, 37] (Fig 1). The following section explores the role of attention, navigation, and imagery in the process of viewing sketches.
Illustration publication number: WW2620UWAC.
The role of attention, navigation, and imagery in viewing sketches
While the core spatial processing network is symmetrically distributed in the dorsal frontal-parietal regions (presupplementary motor area), the anterior insula and frontal operculum of both hemispheres suggest that attention, navigation, and imagery play important roles in sketching [33, 38]. The dorsal frontal-parietal regions are consistently implicated in spatial processing across a range of tasks [39]. The dorsal parietal cortex mediates top-down visual attention and the selection of stimuli, while the ventral frontoparietal network regulates the bottom-up orientation of attention and the task-relevant stimuli of visual perception [40]. The frontal-parietal areas, including both the frontal and parietal lobes, are bilaterally involved in a wide range of cognitive processes, such as attention, working memory, decision-making, and motor control-related spatial information [38]. The parietal cortex is responsible for visual–spatial attention in spatial perception, and in planning and executing movements, which helps designers focus on specific parts of the sketch. Meanwhile, the prefrontal cortex is responsible for interpreting the sketch and making sense of it. On the other hand, the ventral frontoparietal network is responsible for processing object recognition and perception. This network is closely connected with the inferior parietal lobe, which plays a crucial role in visual–spatial processing.
To delve deeper into the underlying mechanisms, it is crucial to consider the role of spatial navigation. Spatial navigation plays a critical role in spatial processing and is closely related to long-term memory. Recent single-cell investigations have identified neurons that regulate and encode spatial properties during navigation, including place cells, grid cells, and head direction cells [41, 42]. Studies have shown that the parahippocampal place area (PPA) is activated when individuals view or imagine scenes or environments. In particular, the PPA is involved in the recognition and processing of spatial information, such as the layout and organization of objects within a scene. This makes the PPA important for visual–spatial thinking and sketching, as it helps an individual to mentally visualize and manipulate spatial information.
While PPA is responsible for recognizing specific views in environment-based navigation, the retrosplenial cortex (RSC) converts allocentric to egocentric representations [43]. The ability to generate mental imagery is critical for sketching, and the RSC is thought to play a role in mental imagery and the construction of spatial maps [44, 45]. Studies have shown that this region is activated when individuals are asked to imagine and mentally manipulate objects in space or navigate through virtual environments. Thus, the RSC is an important component of the brain’s network for spatial cognition and is involved in a range of processes that contribute to visual–spatial thinking and sketching.
As we delve further into cognitive processes, spatial memory emerges as a key player. Spatial memory refers to the ability to remember and mentally manipulate spatial information, which is supported by brain regions, such as the hippocampus [46]. Research have shown that sketching influence the demand for spatial memory in interpreting or remembering specific aspects of design [47]. While hippocampus and other related brain regions are involved in recalling and navigating the physical environment [48]. The prefrontal and anterior cingulate cortex are critical for the storage and retrieval of remote spatial memories involving parietal and RSCs during the consolidation of remote memory [49].
The right hippocampus appears to be particularly involved in memory for locations within an environment, with the left hippocampus being more involved in context-dependent episodic or autobiographical memory [50]. Robin et al. [51] explored how spatial context plays a crucial role in the neural representation of events with functional magnetic resonance imaging (fMRI) and showed that the hippocampus and surrounding areas were more active when events were presented in a spatial context, suggesting that spatial information is prioritized in the neural representation of events. The use of VR on spatial memory is also related to the method of loci by which one links their spatial scenario with objects to increase memory capacity [52].
Spatial ability is influenced by individual factors
Individual factors, such as age, gender, and background, can influence the spatial cognition involved in sketching [53]. Studies have shown that spatial ability tends to decline with age and that there is a gender difference, with men typically outperforming women in spatial tasks. Additionally, research has found that users with different spatial abilities exhibit distinct drawing behaviors in VR, and that spatial ability can impact the shape of the drawing [54]. A person’s spatial ability can be assessed using various methods [55, 56]. Pellegrino (1984) proposed that spatial ability can be broken down into two major factors: spatial relations and spatial visualization ability [57]. The Purdue spatial visualization test (PSVT) is commonly used to evaluate spatial ability, particularly the capacity to mentally rotate 3D objects [58]. The rotation test (PSVT:R) is a validated test for measuring intrinsic and static spatial abilities, including psychovisualization and mental image rotation [59, 60]. It consists of 30 items to be completed within 20 minutes and demonstrates high internal consistency according to the Kuder–Richardson formula. Researchers across various fields have used PSVT:R to measure individual spatial abilities. While, in this study, we did not consider spatial ability a major variable, it was considered a covariate variable.
Materials and methods
This study compared the cognitive effects of using 2D and 3D sketching tools for different tasks. To observe brain activation related to viewing 3D sketching while also avoiding head motion, the experiment was split into two stages: sketching-recording and viewing-scanning. The participants first completed sketching tasks with both 2D and 3D tools in the sketching-recording. Then, during the viewing-scanning stage, which was conducted separately to avoid head motions, they viewed their sketches while their brain activity was recorded with fMRI. The results were then compared to determine whether there were any noticeable differences in brain activation either when viewing 2D and 3D sketching environments or between the different sketching tasks.
Experimental design
All participants were asked to complete the sketching-recording stage using each setup sequentially, starting with the 2D tool and then moving on to the 3D tool. Prior to the start of the first stage, participants received comprehensive information about the study and provided their consent by signing a consent form.
The sketching-recording stage involved 1) a spatial task, 2) a tutorial exercise, and 3) six recording tasks. All participants were asked to complete the first stage before starting the second stage. The tutorial exercise lasted approximately 15 minutes and was designed to ensure that participants understood how to use both tools. During the tutorial exercise, the experimenter introduced the 2D tool with manual instructions and asked the participants to draw horizontal lines, a cube, and an object on the sketching board using a stylus. Printed images were provided to help clarify the criteria. Next, the participants were asked to use the VR device and the right-hand controller to draw horizontal lines, a cube, and an object perspective in the sketching space. Printed images were also provided to clarify the criteria. Participants were permitted to take as much time as they needed during the tutorial exercise to become familiar with both setups. All participants successfully completed the exercise within 15 minutes. Once the participants were familiar with both tools, the experiment began.
All participants completed six tasks in sequence using the 2D and 3D tools provided in the experiment. The participants were asked to draw, using each tool, the following three elements: lines, a geometrical object (the cube), and a naturalistic object (a tree perspective). Starting with the 2D tool, the tasks were administered in a specific sequence, and participants performed the line task, followed by the geometrical object task and, finally, the naturalistic object task. Subsequently, participants proceeded to the 3D tool and performed the same tasks in the same order (see Fig 2). To ensure that the participants focused on the sketching process, the sketching time for each task was limited to 30 seconds. The 3D setup included a 2019 wireless head-mounted Oculus Quest VR display (128 GB and 1440 × 1600 pixels per eye at 72 Hz) and a virtual two-handed controller for use with the Tilt Brush 3D drawing application (Google Co., 2006). The 2D drawing setup included a 12.9-inch iPad Pro and Apple Pencil (2018) and the Adobe Photoshop application, which has a functional layout similar to Tilt Brush. Both setups were set to an empty sketching canvas with a white background, and the only available stroke color was black.
Following the sketching-recording stage, the experimental design progressed to the viewing-scanning stage. During this stage, the participants watched the video recordings of their sketching tasks while being monitored by a Siemens 3T MAGNETOM Prisma magnetic resonance imaging (MRI) platform at the Image Center for Integrated Body, Mind, and Culture Research at National Taiwan University. All six sketching videos completed during the first stage were viewed by the participants in the second stage.
We pseudorandomly assigned the order in which the participants viewed the videos in three repeated sequences to ensure robust responses. Each sequence included three tasks performed using both sketching tools, resulting in a total of 18 videos (3 tasks × 2 sketching tools × 3 times), as depicted in Fig 3. The tasks were presented using a block design with interleaved trials to avoid consecutive presentations of the same tool type. This approach maximized the statistical power and created a different spatial sketching experience for the participants.
Participants and ethics
The study recruited 20 right-handed students (10 female) from the Department of Horticulture and Landscape Architecture (mean age = 24.4 years; SD = 2.23 years). All participants had normal or corrected-to-normal vision, at least three years of landscape sketching experience, and formal training by the university. None of the participants claimed a history of psychological, physical, or neural symptoms or injury.
The Behavioral and Social Science Research Ethics Committee at National Taiwan University approved the study, and all participants provided informed consent by signing the consent form (Ethical Review Approval No.: 201908HM003). Each participant received NT$500 (approximately USD$17) as monetary compensation for their participation in the study.
Spatial ability tasks as cognitive controls
Spatial ability is the capacity to mentally represent, manipulate, and comprehend the relationship between object positions within a visual stimuli [55, 56]. To address the potential confounding factor of spatial ability, this study utilized PSVT, as recommended by previous studies [59, 60]. Data were also gathered to test whether gender and age affected spatial ability. The study examined the effects of age and gender on spatial ability by conducting a two-way analysis of variance (ANOVA), which considered the impact of both independent variables and their interactions on spatial ability scores. The results of the analysis are presented in Table 1 and reveal no significant interaction between age, sex, and spatial ability scores, which indicates that the effects of these variables on the dependent variable were not interdependent (see Table 1).
We further examined the main effect of gender and age on spatial ability, and the results showed no significant difference in spatial ability test scores between males (mean [SD], 15.4 [2.83]) and females (mean [SD], 14.6 [3.72]); t(18) = 0.54; p = 0.59; d = 0.24 (see Table 2) or based on age (F(5,12) = 1.256, p = 0.347) (see Table 3).
MRI setup acquisition and data analysis
The recorded stimuli videos were displayed on the screen of an MR-compatible image interaction platform using EPrime 2.0 software. MR images were acquired using a Siemens 3T MAGNETOM Prisma MRI scanner with a 32-channel head coil.
The T1-weighted high-resolution images were acquired at the following settings: TE(echo time) = 2.3 ms, TR(repetition time) = 2000 ms, FA = 8°, matrix size = 256 × 256 × 192, FOV(field-of-view) = 240 × 240 mm2, voxel size = 0.93 × 0.93 × 0.93 mm3, a bandwidth of 200 Hz/pixel, and number 192 slices. We obtained the T2-weighted images at the following settings: TE = 103 ms, TR = 953 ms, FA = 150°, matrix size = 288 × 384 × 045, FOV = 192 × 192 mm2, voxel size = 0.66 × 0.50 × 3.00 mm3, and number of slices = 45. This was followed by the 2D echo-planar image sequences acquired through 252 scans, with TE = 30 ms, TR = 3000 ms, FA = 90°, matrix size = 64 × 64 × 45, FOV = 192 × 192 mm2, voxel size = 3.00 × 3.00 × 3.00 mm3, and a bandwidth of 2440 Hz/pixel (interleaved slice order/ bottom up).
Preprocessing.
All the Digital Imaging and Communications in Medicine (DICOM) images were first reconstructed into the NIfTI-1 format (.nii) using DICOM import in Statistical Parametric Mapping tool version 12 (SPM12) [61]. Subsequently, we performed the following preprocessing procedure. First, we conducted motion correction (realign and unwarp) to account for motion artifacts in the fMRI time series. This involved realigning all Echo Planar Imaging (EPI) scans (each with 252 slices) to correct for translational and rotational motion for each participant. We used a Gaussian smoothing kernel with a full width at half maximum (FWHM) of 5 mm. The images were then resliced to match the mean of the image. No wrapping or weighting was applied. We checked the translation and rotation of the x, y, and z axes of the head center for each participant. According to the SPM list, the acceptable translation and rotation ranges are both within ± 5 mm. For the present experiment, translation and rotation within the x, y, and z axes were checked, with acceptable ranges set at ± 3 mm. Second, we conducted slice timing correction to account for differences in image acquisition time between slices. The fMRI data consisted of 45 slices per volume with TR = 3000 ms. The reference slice was set as the middle slice (23), and the slice order was bottom up and interleaved.
Third, we coregistered the high-resolution anatomical image to the functional images using coregistration algorithms. This step improved the alignment of images with the Montreal Neurological Institute (MNI) template. We used the mean EPI image as the reference image and the T2-weighted image as the source image, as they shared the same slice sequence and number. The objective function was set as normalized mutual information with default separation and tolerances according to SPM 12 settings.
Fourth, to enhance the interpretation of the brain images, we applied segmentation to separate the images into cerebrospinal fluid, white matter, and gray matter for easier interpretation. We saved bias-corrected images to ensure uniform intensities within different tissue types. We affined regularization to the (International Consortium for Brain Mapping (ICBM) space template of East Asian brains for local optimization, and forward deformation was used to normalize the images to the MNI space.
Fifth, we performed normalization, and a more accurate analysis of individual brain images was modified into brain templates (bounding box = [−78 −112 −70 78 76 85]) with a default voxel size [3 3 3]. Finally, spatial smoothing was applied to improve the signal-to-noise ratio. We used a filter kernel with an FWHM of [6, 6, 6], which is twice the voxel size, effectively strengthening the signal and reducing noise.
fMRI data analysis.
To examine the brain scans of individual participants while they viewed various sketches, we conducted a first-level analysis. This analysis employed a t-test using the SPM Menu protocol and considered six distinct conditions. Each condition involved a specific video block design lasting 30 seconds, interspersed with resting intervals of 12 seconds. Conditions 1–3 used 2D tools for sketching lines, geometric objects, and natural objects, respectively. Conditions 4–6 employed 3D tools for sketching lines, geometric objects, and natural objects, respectively. These conditions were presented in a pseudorandom sequence within a single session. The general linear model (GLM) was applied to measure the blood oxygenation level dependent (BOLD) signal and estimation. The initiation of each block was determined at a precise time point as follows: Condition 1 = [172 s, 424 s, 718 s], Condition 2 = [4 s, 298 s, 592 s], Condition 3 = [130 s, 256 s, 676 s], Condition 4 = [46 s, 382 s, 508 s], Condition 5 = [88 s, 466 s, 550 s], and Condition 6 = [214 s, 340 s, 634 s]. The realignment parameter was also applied as a regressor to eliminate head motion potential. No predefined masks were chosen at the p-value of FWE = 0.05 at the extent threshold of > 39 voxels according to the SPM cluster size threshold calculation [62]. The single-subject results were visualized using the xjView toolbox (https://www.alivelearn.net/xjview).
For the group statistical inference, we conducted groupwise analysis using the SPM tool to specify the second-level analysis with the 2 × 3 flexible factorial design [2 (2D and 3D) × 3 (line, geometrical object, naturalistic object)] as coded (A1 = non-VR, A2 = VR, B1 = line, B2 = cubic, B3 = object). The main effects compared the non-VR (2D) and VR (3D) conditions with the consideration of spatial ability as a covariate variable. Furthermore, we examined the interactions among different tools across three conditions, enhancing our comprehension of how independent variables relate to differences in brain activation.
Results
The goal of the experiment was to assess whether there were noticeable differences in brain activation when participants viewed 2D and 3D sketching environments. Prior to this, we conducted an initial check to confirm that the participants’ spatial abilities were consistent across age and gender groups. Furthermore, we categorized the sketching scenarios into three distinct tasks. The results of our analysis unequivocally revealed substantial differences in neural activation among participants when they observed 2D sketching processes in contrast to 3D sketching processes. Additionally, we conducted further comparisons between 2D and 3D sketching environments for tasks involving line drawing, geometric objects, and naturalistic objects. These comparisons also highlighted significant differences in brain activation, which will be elaborated on in the subsequent section.
Comparison of 2D and 3D sketching environments
This section compares the participants’ brain activation patterns during their exposure to the sketching videos in both 2D and 3D environments. It examines any noteworthy differences in brain activity between these two settings to provide a comprehensive overview of the data analysis across all trials. As shown in Table 4, when comparing the views of the 2D and 3D sketching processes, significant differences were found on the right middle temporal gyrus (MTG), left middle occipital gyrus (MOG), left postcentral gyrus, and right subgyral (PFWE < 0.05, T = 5.07).
Fig 4 shows the differences observed in cognitive function and brain activation when participants viewed the 3D sketching environment. These results support our research hypothesis that 3D sketching media stimulate different activations of brain activity compared to 2D sketching media.
Viewing line sketches in 2D and 3D environments
We further focused on examining brain activation differences during the viewing of the line sketches in both 2D and 3D environments, investigating any distinctive findings. First, the results of the line sketching tasks (comparing Condition 1 to Condition 4) showed significant activation of the right MTG and the left MOG when viewing the 3D sketching environment (Table 5 and Fig 5).
Viewing geometrical object sketches in 2D and 3D environments
While observing the sketching process of geometric objects in both 2D and 3D environments, significant activations were identified in the right MTG and the left cuneus (Table 6). These activations were particularly pronounced when comparing the cube sketching process between the 2D and 3D environments (PFWE < 0.05, T = 6.74) (Fig 6).
Viewing naturalistic object sketches in 2D and 3D environments
Next, we turn our attention to the comparison of naturalistic object sketches in both 2D and 3D environments. As shown in Table 7, we observed noteworthy activations in the bilateral cuneus, the MOG, and the right side of the lingual gyrus (PFWE < 0.05, T = 6.74) in the 3D condition. This finding underscores a substantial difference in brain activations when participants view naturalistic object sketches in 2D versus 3D environments (see Fig 7).
Discussion and conclusions
In this study, we investigated the differences in brain activation patterns that occurred when the participants viewed sketching processes in both 2D and 3D environments across various sketching tasks. Our findings unveiled significant disparities in brain activity when participants experienced sketching in 3D versus 2D environments. The disparities were particularly apparent in regions such as the right MTG, left MOG, left postcentral gyrus, and right parietal lobe. These observations revealed the intricate interplay between visual, spatial, and motor brain activation associated with 3D sketching tools, specifically in the temporal and occipital gyrus regions. Furthermore, a deeper examination of different sketching conditions revealed nuanced variations in neural activation. Specifically, under the line condition, significant activations were observed in the right MTG and left MOG in the 3D environment as opposed to the lack of activation observed in these areas under the same condition in the 2D condition. Meanwhile, the geometrical condition and the naturalistic condition both yielded activations in the cuneus. These observations offer a more detailed understanding of how sketching with different content in a 3D sketching environment elicits different neural activations during visual and spatial processes as compared to 2D sketching environment.
The activation of the right MTG is known for its role in semantic memory processing, multimodal sensory integration, and spatial processing [63]. Furthermore, research has shown that MTG is necessary for forming nonsalient, novel, and creative insights [64]. Our findings suggest that viewing linear and geometrical object sketches in 3D environments may involve similar brain activation, while 3D freehand sketching is less related to accuracy than to the relationships between shapes and space.
The MOG is involved in early visual processing and is associated with basic visual perception [65]. During the line condition and naturalistic object condition, but not in the geometrical object condition, the MOG played a critical role when the participants viewed the sketch in a 3D environment. While the MOG was more pronounced when viewing natural objects in 3D than when viewing them in 2D, it was still involved significantly in the spatial tasks, such as sketching in the virtual space. Furthermore, the cuneus and the lingual gyrus, which are involved in basic visual–spatial processing [66], were also critical areas in 3D object sketching but not in 3D line sketching. While the cuneus is known to receive basic visual–spatial information from the retina [67], our study showed that the 3D sketching environment of both geometrical and naturalistic objects demands higher visual–spatial cognition.
The parietal lobe is crucial for spatial tracking [68], depth perception [69], stereopsis [70], and egocentric spatial representation in the dorsal pathway. Activation of the postcentral gyrus may also be linked to the feeling of presence and vividness or interactivity experienced by an individual in a 3D sketching environment [71]. These regions emphasize the complexity of 3D sketching processes and their connection to visual–spatial cognition functions. Contrary to our expectations, across all conditions, the PPA and RSC did not have a significant impact on the 3D sketching environment compared to the 2D environment. According to previous findings [72], we considered that this may be related to the fact that our sketching space setting did not provide enough scene features and elements.
Given the immersive and navigational aspects of 3D sketching tools, they have great potential to enhance spatial experiences and sensory engagement in 3D environments. Our study answers the research question by demonstrating that viewing sketching processes in 2D and 3D environments results in different visual and spatial brain activations. Moreover, our study reaffirms that different sketching objects in a 3D sketching setting demand higher visual and spatial cognition. These findings emphasize the role of 3D spatial sketching tools and highlight their ability to activate specific neural processes.
Furthermore, our findings offer novel insights into the impact of a 3D tool on creative spatial thinking, exemplified by a comparison to freehand sketching. Previous literature has discussed how sketching can be used as a thinking tool to externalize concepts, explore ideas, and solve problems [73]. We further extend our horizon to the concept of “R-mode thinking,” which refers to a creative, holistic, and intuitive way of thinking [74]. When individuals sketch in a 3D sketching environment, they may enhance their ability to perceive and understand the spatial relationship between immersion and the feeling of presence on a different scale. When tapping into their “R-mode thinking,” it is clear that the thinking focus shifts away from precise details and more toward visual and spatial perceptions. This finding implies that the “R-mode” system of thinking plays a significant role when individuals work with 3D sketching tools. Future research could aim to establish behavioral correlations with neural activity to explore how sketching in a 3D environment might elicit R-mode thinking during sketching processes, thus affecting spatial perception and creativity in design.
While our study provides valuable insights, it has limitations that require acknowledgment. To enhance the applicability of our findings, future research should use upgraded hardware and diverse experimental designs, including behavioral data. Additionally, our study mainly focused on landscape design students. Expanding the research to untrained individuals or other academic disciplines can broaden its relevance. Future research should also delve into the intricate relationship between brain activity and behavior, especially in the context of spatial thinking, to reveal practical implications. Furthermore, exploring the potential of immersive VR technologies for creating lifelike representations of complex structures holds promise for future studies in this field.
In conclusion, our study underscores the cognitive advantages associated with the use of 3D VR sketching tools, which elicit enhanced visual and spatial brain activation compared to traditional 2D tools. These insights hold substantial promise for advancing design thinking and have implications for future developments in spatial thinking and VR-based design sketching. Furthermore, our study provides neural evidence of brain activation during 3D sketching, as depicted in Fig 8. Moving forward, it is imperative for future research to establish links between this neural activity and behavioral outcomes, thereby facilitating a more comprehensive understanding of the impact of 3D sketching on cognitive processes.
Illustration publication number: IN2620T6E1.
Acknowledgments
We are grateful to all the participants for the effort and time they devoted to the experiment. Special thanks to the team members at the Imaging Center for Integrated Body, Mind, and Culture Research at National Taiwan University for assisting with the fMRI investigation and for their contributions to the present analysis.
References
- 1. Aristotle , Lawson-Tancred H. De Anima (On the Soul): Penguin Publishing Group; 1986.
- 2. Goldschmidt G. The dialectics of sketching. Creativity research journal. 1991 Jan 1;4(2):123–43.
- 3. Schon DA, Wiggins G. Kinds of seeing and their functions in designing. Design studies. 1992 Apr 1;13(2):135–56.
- 4.
Goldschmidt G. Design representation: Private process, public image. InDesign representation 2004 (pp. 203–217). London: Springer London.
- 5. Goel V. Sketches of thought: MIT Press; 1995.
- 6. Zeleznik RC, Herndon KP, Hughes JF. SKETCH: An interface for sketching 3D scenes. InACM SIGGRAPH 2006 Courses 2006 Jul 30 (pp. 9-es).
- 7. Wu JC, Chen CC, Chen HC. Comparison of Designer’s Design Thinking Modes in Digital and Traditional Sketches. Design and Technology Education. 2012;17(3):37–48.
- 8. Li R, Xu D. Distribution of landscape architecture based on 3D images and virtual reality rationality study. IEEE access. 2020 Jul 17;8:140161–140170.
- 9. Gutierrez JM, Dominguez MG, Gonzalez CR. Using 3D virtual technologies to train spatia lskills in engineering. The International journal of engineering education. 2015;31(1):323–334.
- 10. Rafi A, Anuar K, Samad A, Hayati M, Mahadzir M. Improving spatial ability using a Web-based Virtual Environment (WbVE). Automation in construction. 2005 Dec 1;14(6):707–15.
- 11. Steuer J, Biocca F, Levy MR. Defining virtual reality: Dimensions determining telepresence. Communication in the age of virtual reality. 1995 Jun 1;33:37–39.
- 12. Yang EK, Lee JH. Cognitive impact of virtual reality sketching on designers’ concept generation. Digital Creativity. 2020 Apr 2;31(2):82–97.
- 13. Wang P, Wu P, Chi HL, Li X. Adopting lean thinking in virtual reality-based personalized operation training using value stream mapping. Automation in Construction. 2020 Nov 1;119:103355.
- 14. Liu Y, Castronovo F, Messner J, Leicht R. Evaluating the impact of virtual reality on design review meetings. Journal of Computing in Civil Engineering. 2020 Jan 1;34(1):04019045.
- 15. Martín-Gutiérrez J, Mora CE, Añorbe-Díaz B, González-Marrero A. Virtual technologies trends in education. Eurasia journal of mathematics, science and technology education. 2017 Jan 18;13(2):469–486.
- 16.
Burdea GC, Coiffet P. Virtual reality technology. John Wiley & Sons; 2003 Jun 30.
- 17. Howard-Jones P, Ott M, van Leeuwen T, De Smedt B. The potential relevance of cognitive neuroscience for the development and use of technology-enhanced learning. Learning, media and technology. 2015 Apr 3;40(2):131–151.
- 18. Wenk N, Penalver-Andres J, Buetler KA, Nef T, Müri RM, Marchal-Crespo L. Effect of immersive visualization technologies on cognitive load, motivation, usability, and embodiment. Virtual Reality. 2023 Mar;27(1):307–331. pmid:36915633
- 19. Paes D, Irizarry J, Pujoni D. An evidence of cognitive benefits from immersive design review: Comparing three-dimensional perception and presence between immersive and non-immersive virtual environments. Automation in Construction. 2021 Oct 1;130:103849.
- 20. Beck L, Wolter M, Mungard NF, Vohn R, Staedtgen M, Kuhlen T, et al. Evaluation of spatial processing in virtual reality using functional magnetic resonance imaging (fMRI). Cyberpsychology, Behavior, and Social Networking. 2010 Apr 1;13(2):211–215. pmid:20528281
- 21. Merchant Z, Goetz ET, Keeney-Kennicutt W, Kwok OM, Cifuentes L, Davis TJ. The learner characteristics, features of desktop 3D virtual reality environments, and college chemistry instruction: A structural equation modeling analysis. Computers & Education. 2012 Sep 1;59(2):551–568.
- 22. Glick S, Porter D, Smith C. Student visualization: Using 3-D models in undergraduate construction management education. International Journal of Construction Education and Research. 2012 Jan 1;8(1):26–46.
- 23. Reggente N, Essoe JK, Aghajan ZM, Tavakoli AV, McGuire JF, Suthana NA, et al. Enhancing the ecological validity of fMRI memory research using virtual reality. Frontiers in neuroscience. 2018 Jun 15;12:408. pmid:29962932
- 24. Paes D, Arantes E, Irizarry J. Immersive environment for improving the understanding of architectural 3D models: Comparing user spatial perception between immersive and traditional virtual reality systems. automation in Construction. 2017 Dec 1;84:292–303.
- 25. Freina L, Ott M. A literature review on immersive virtual reality in education: state of the art and perspectives. InThe international scientific conference elearning and software for education 2015 Apr 23 (Vol. 1, No. 133, pp. 10–1007).
- 26. Bach B, Sicat R, Beyer J, Cordeil M, Pfister H. The hologram in my hand: How effective is interactive exploration of 3D visualizations in immersive tangible augmented reality?. IEEE transactions on visualization and computer graphics. 2017 Aug 29;24(1):457–467. pmid:28866590
- 27. Alatta RA, Freewan A. Investigating the effect of employing immersive virtual environment on enhancing spatial perception within design process. ArchNet-IJAR: International Journal of Architectural Research. 2017 Jul 1;11(2):219.
- 28. Miller JA, Khatib F, Hammond H, Cooper S, Horowitz S. Introducing Foldit education mode. Nature structural & molecular biology. 2020 Sep;27(9):769–70. pmid:32807992
- 29. Oti A, Crilly N. Immersive 3D sketching tools: Implications for visual thinking and communication. Computers & Graphics. 2021 Feb 1;94:111–123.
- 30. Chatterjee A. The neural organization of spatial thought and language. InSeminars in speech and language 2008 Aug (Vol. 29, No. 03, pp. 226–238). pmid:18720319
- 31. Palmer S. Fundamental aspects of cognitive representation. In: Rosch E, Lloyd B, editors. Cognition and Categorization: Lawrence Elbaum Associates; 1978. p. 259–303.
- 32.
Newcombe NS, Shipley TF. Thinking about spatial thinking: New typology, new assessments. InStudying visual and spatial reasoning for design creativity 2014 Nov 9 (pp. 179–192). Dordrecht: Springer Netherlands.
- 33. Cona G, Scarpazza C. Where is the “where” in the brain? A meta‐analysis of neuroimaging studies on spatial cognition. Human brain mapping. 2019 Apr 15;40(6):1867–1886. pmid:30600568
- 34. Mishkin M, Ungerleider LG, Macko KA. Object vision and spatial vision: two cortical pathways. Trends in neurosciences. 1983 Jan 1;6:414–417.
- 35. Biederman I, Vessel EA. Perceptual pleasure and the brain: A novel theory explains why the brain craves information and seeks it through the senses. American scientist. 2006 May 1;94(3):247–253.
- 36. Levy R, Goldman-Rakic PS. Segregation of working memory functions within the dorsolateral prefrontal cortex. Executive control and the frontal lobe: Current issues: Springer; 2000. p. 23–32. pmid:10933207
- 37. Ayzenberg V, Simmons C, Behrmann M. Temporal asymmetries and interactions between dorsal and ventral visual pathways during object recognition. Cerebral Cortex Communications. 2023;4(1):tgad003. pmid:36726794
- 38. Chica AB, Bartolomeo P, Lupiáñez J. Two cognitive and neural systems for endogenous and exogenous spatial attention. Behavioural brain research. 2013 Jan 15;237:107–123. pmid:23000534
- 39. Sereno MI, Huang RS. Multisensory maps in parietal cortex. Current opinion in neurobiology. 2014 Feb 1;24:39–46. pmid:24492077
- 40. Corbetta M, Miezin FM, Shulman GL, Petersen SE. A PET study of visuospatial attention. Journal of Neuroscience. 1993 Mar 1;13(3):1202–1226. pmid:8441008
- 41. Spiers HJ, Barry C. Neural systems supporting navigation. Current opinion in behavioral sciences. 2015 Feb 1;1:47–55.
- 42. Shelton AL, Gabrieli JD. Neural correlates of encoding space from route and survey perspectives. Journal of Neuroscience. 2002 Apr 1;22(7):2711–217. pmid:11923436
- 43. Epstein RA. Parahippocampal and retrosplenial contributions to human spatial navigation. Trends in cognitive sciences. 2008 Oct 1;12(10):388–396. pmid:18760955
- 44. Miller AM, Vedder LC, Law LM, Smith DM. Cues, context, and long-term memory: the role of the retrosplenial cortex in spatial cognition. Frontiers in human neuroscience. 2014 Aug 5;8:586. pmid:25140141
- 45. Auger SD, Maguire EA. Retrosplenial cortex indexes stability beyond the spatial domain. Journal of Neuroscience. 2018 Feb 7;38(6):1472–81. pmid:29311139
- 46. Dupret D, O’neill J, Pleydell-Bouverie B, Csicsvari J. The reorganization and reactivation of hippocampal maps predict spatial memory performance. Nature neuroscience. 2010 Aug;13(8):995–1002. pmid:20639874
- 47. Bilda ZA, Gero JS. Does sketching off-load visuo-spatial working memory. Studying designers. 2005;5(2005):145–160.
- 48. Eichenbaum H, Cohen NJ. Can we reconcile the declarative memory and spatial navigation views on hippocampal function?. Neuron. 2014 Aug 20;83(4):764–70. pmid:25144874
- 49. Maviel T, Durkin TP, Menzaghi F, Bontempi B. Sites of neocortical reorganization critical for remote spatial memory. Science. 2004 Jul 2;305(5680):96–99. pmid:15232109
- 50. Burgess N, Maguire EA, O’Keefe J. The human hippocampus and spatial and episodic memory. Neuron. 2002 Aug 15;35(4):625–641. pmid:12194864
- 51. Robin J, Buchsbaum BR, Moscovitch M. The primacy of spatial context in the neural representation of events. Journal of Neuroscience. 2018 Mar 14;38(11):2755–2765. pmid:29440386
- 52. Reggente N, Essoe JK, Baek HY, Rissman J. The method of loci in virtual reality: explicit binding of objects to spatial contexts enhances subsequent memory recall. Journal of Cognitive Enhancement. 2020 Mar;4:12–30.
- 53. Sanjuán JM, León C, Gordo JF. Factors influencing spatial skills development of engineering students. The International journal of engineering education. 2017;33(2):680–692.
- 54.
Barrera Machuca MD, Stuerzlinger W, Asente P. The effect of spatial ability on immersive 3d drawing. InProceedings of the 2019 on Creativity and Cognition 2019 Jun 13 (pp. 173–186).
- 55. Piaget J, Inhelder B. The Child’s Conception of Space: Routledge; 1997.
- 56. Gardner H. Multiple intelligences: Minnesota Center for Arts Education; 1992.
- 57. Pellegrino JW, Alderton DL, Shute VJ. Understanding spatial ability. Educational psychologist. 1984 Sep 1;19(4):239–53.
- 58.
Guay R, Purdue Research F, Educational Testing Service Test C. Purdue spatial visualization test. [West Layfette, Ind.]: Purdue University [West Layfette, Ind.]; 1976.
- 59. Bodner GM, Guay RB. The Purdue visualization of rotations test. The chemical educator. 1997 Oct;2(4):1–7.
- 60. Branoff TJ. Spatial visualization measurement: A modification of the Purdue Spatial Visualization Test—visualization of rotations. Engineering Design Graphics Journal. 2009;64(2).
- 61. Penny WD, Friston KJ, Ashburner JT, Kiebel SJ, Nichols TE. Statistical parametric mapping: the analysis of functional brain images: Elsevier; 2011.
- 62. Phillips C. CyclotronResearchCentre, SPM Cluster Size Threshold. (v1.0) ed. Zenodo, 2018.
- 63. Cabeza R, Nyberg L. Imaging cognition II: An empirical review of 275 PET and fMRI studies. Journal of cognitive neuroscience. 2000 Jan 1;12(1):1–47. pmid:10769304
- 64. Shen W, Yuan Y, Liu C, Luo J. The roles of the temporal lobe in creative insight: an integrated review. Thinking & Reasoning. 2017 Oct 2;23(4):321–375.
- 65. Grill-Spector K, Malach R. The human visual cortex. Annu. Rev. Neurosci. 2004 Jul 21;27:649–77. pmid:15217346
- 66. Palejwala AH, Dadario NB, Young IM, O’Connor K, Briggs RG, Conner AK, et al. Anatomy and white matter connections of the lingual gyrus and cuneus. World neurosurgery. 2021 Jul 1;151:e426–37. pmid:33894399
- 67.
Ward J. The student’s guide to cognitive neuroscience. Routledge; 2019 Dec 6.
- 68. Culham JC, Kanwisher NG. Neuroimaging of cognitive functions in human parietal cortex. Current opinion in neurobiology. 2001 Apr 1;11(2):157–163. pmid:11301234
- 69. Breveglieri R, Borgomaneri S, Filippini M, De Vitis M, Tessari A, Fattori P. Functional connectivity at rest between the human medial posterior parietal cortex and the primary motor cortex detected by paired-pulse transcranial magnetic stimulation. Brain Sciences. 2021 Oct 15;11(10):1357. pmid:34679421
- 70. Sack AT. Parietal cortex and spatial cognition. Behavioural brain research. 2009 Sep 14;202(2):153–161. pmid:19463696
- 71. Iachini T, Maffei L, Masullo M, Senese VP, Rapuano M, Pascale A, et al. The experience of virtual reality: are individual differences in mental imagery associated with sense of presence?. Cognitive processing. 2019 Aug 1;20(3):291–298. pmid:30569268
- 72. Köhler S, Crane J, Milner B. Differential contributions of the parahippocampal place area and the anterior hippocampus to human memory for scenes. Hippocampus. 2002;12(6):718–723. pmid:12542224
- 73. Leblanc T. Sketching as a thinking process. InDS 82: Proceedings of the 17th International Conference on Engineering and Product Design Education (E&PDE15), Great Expectations: Design Teaching, Research & Enterprise, Loughborough, UK, 03–04.09. 2015 2015 (pp. 606–611).
- 74. Edwards B. Drawing on the Right Side of the Brain. InCHI’97 Extended Abstracts on Human Factors in Computing Systems 1997 Mar 22 (pp. 188–189).