Definitions and components of creativity

Theories of creativity reach back thousands of years. Acknowledging differences in thought processes was present in the culture of Ancient Greece [17], though people also “believed that creativity required the intervention of the muses” [18, p. 152] and that artists were not considered creative in and of themselves. The earliest empirical inquiry into human creative behavior can be traced to Galton’s [19] work on human capabilities. In more recent decades, defining creativity has been perhaps the most central topic and challenge in contemporary creativity research [20]. Guilford [1] was an early champion of the importance of scientific research in creativity, yet even he seemed to struggle defining what creativity is; “In its narrow sense, creativity refers to the abilities that are most characteristic of creative people” (p. 444). Runco and Jaeger [20] contended that the first clear scholarly description of creativity was offered by Stein [21], who defined creativity as, “a novel work that is accepted as tenable or useful or satisfying by a group in some point in time” (p. 311). Stein’s [21] definition has been adopted by many researchers [22–24] and includes two important criteria: originality and usefulness.

Originality refers to an idea or product not having previously existed. This is crucial to the construct of creativity. The ability to convincingly forge the Mona Lisa may require talent, but it does not necessarily mean a person is creative. Creativity requires novelty. However, many “ideas and products that are merely original might very well be useless” [20, p. 93], and so novelty is a necessary but not sufficient qualification for creativity. Therefore, for products to qualify as creative, they must be original and useful.

The dual criteria of originality and usefulness have been troublesome for creativity researchers, primarily because judging originality and usefulness involves some subjectivity [25, 26]. The novelty required for a product to qualify as creative could theoretically be measured objectively, but this would require accurately determining whether a product has previously existed somewhere in the world at some point in time. To prove that an idea is truly novel and has never existed before is inherently illogical because it requires proving a negative. Usefulness is even more subjective because it relies almost entirely on context. This usefulness is often determined by the community or group that the product is created for or by individuals who encounter the idea or product. Runco et al. [27] postulated that from this perspective, “creativity can lead to things which are original and useful but only for the individual creator himself or herself” [27, p. 366]. On the other hand, Corazza’s [28] dynamic definition of creativity, recognizes that value judgements of usefulness can be mistaken and is useful for understanding creativity as a process that occurs within an environmental, cultural, and individual context. Thus, the subjectivity of judgements of creative accomplishments is not a problem in this viewpoint and is simply part of the challenge of bringing creative ideas to fruition—and a topic worthy of study in its own right.

Non-cognitive influences on creativity

Although Stein’s [21] requirements that creative thought results in original and useful products, others have expanded theories of creativity to include other behaviors and characteristics. Another important facet of creativity is the influence of non-cognitive elements like personality and motivation on its expression [22]. For example, Furnham and Bachtiar [29] and Furnham and Nederstrom [30] found that personality trait extroversion correlated positively with test scores of divergent thinking. Similarly, Wang et al. [31] found that extroversion correlated substantially with scholarly creative achievement in undergraduate students. Openness to experience has also been shown to relate to creative expression. Benedek, et al. [32] found that jazz musicians scored higher on tests of divergent thinking and openness to experience compared to their folk and classically trained counterparts. Fink and Woschnjak [33] found a similar pattern in dancers; modern/contemporary performers scored higher in creativity and openness to experience than individuals trained in theater and ballet. In addition to the direct relationship between openness and creativity, openness mediates the relationships between temperament variables and creativity [34]. Temperament can also directly influence creative expression. For example, Nęcka and Hlawacz [35] found the temperament trait activity correlated positively with creativity, while emotional reactivity was negatively correlated.

Motivation also likely influences creative expression. Hannam and Narayan [36] found that university students were more likely to produce creative work if they were intrinsically motivated and perceived the work environment as fair. Saether [37] also discovered that motivation mediated between creativity and fairness in a sample of online survey responses. These results seem in line with Silvia et al.’s [38] assertion that motivation and its effect on creativity includes aspects of, “goals, self-regulatory processes, and experiences that foster or impede wanting to invest time in creative activities” (p. 114).

Moreover, it is possible that many individuals possess the capacity to be creative but choose not to do so [23]. Torrance acknowledged that scores from creativity tests do not ensure creative accomplishment [27]. Because of its multifaceted nature, with roots in personality, the social environment, and cognition, Mumford and Gustavson [39] have argued that creativity ought to be considered the product of a system of characteristics, rather than a single, isolatable characteristic.

Thus, creativity is a complex, multifaceted construct. However, there are commonalities among the dueling definitions. Weiss et al. [13] found that, across definitions, idea generation (often called fluency) and originality are part of nearly every scholarly definition of creativity. These two aspects of creative behavior are well captured in the concept of divergent thinking, which is the focus of our scholarly investigation.

Importance of creativity

Creativity represents one of the pinnacles of human experience. Indeed, it is difficult to imagine human progress in any of its forms without the catalyzing spark and sustaining force of creativity. Vygotsky held that, “creativity is an essential condition for existence and all that goes beyond the rut of routine” [40, p. 11]. This alone makes creativity worthy of research, but some theorists have proposed other reasons.

One reason to empirically study creativity is its role in fostering economic growth. There is a major economic need for domestic and international industries to quickly and efficiently solve problems, and many simple jobs that need little to no creative output are being automated [41]. It will become increasingly important for the advancing global economy to foster and identify creativity in all age groups [39]. Creativity is crucial in educational organizations as well. In a review of the empirical creativity literature, Davies et al. [42] found evidence that creative learning environments positively impacted student academic attainment, concentration, enjoyment, enthusiasm, and emotional development.

Measuring creativity with the torrance tests of creative thinking

An essential requirement for the empirical study of creativity is a method of measuring creative behavior. The most widely used measures of creativity are the Torrance Tests of Creative Thinking (TTCT), which consist of a Verbal test reliant on written linguistic responses and a non-verbal Figural test that uses pictorial stimuli and requires responses that the examinee must draw (see detailed description below). However, the TTCT does not measure the totality of creative thinking; instead, it measures one of the building blocks of creative behavior: divergent thinking, i.e., the capacity to produce a variety of ideas in response to a stimulus [43]. The TTCT measures divergent thinking through fluency (i.e., the number of responses generated), originality (i.e., how much the responses differ from common responses given in the norm sample), and flexibility (which is the variety of types of responses that examinees give). Divergent thinking has been shown to positively correlate with individual creative achievement, most notably in a 50-year longitudinal study where TTCT fluency, flexibility, originality, and elaboration scores all correlated r = .20 to .29 with personal achievements 50 years later, whereas IQ scores had a slight negative correlation with later achievement. [27] Indeed, divergent thinking is so important to creativity that some researchers claim that divergent thinking is the most valid way to predict creativity, almost using the two terms interchangeably [44]. However, others deny these claims and hold that divergent thinking is only a predictor of individual creativity [29, 45].

Despite the popularity of the TTCT, there remain uncertainties about some of the tests’ psychometric properties. One of the questions regarding the TTCT is the dimensionality and factor structure of its scores. Multiple studies have suggested that the TTCT-F and TTCT-V are multidimensional and follow a two-factor model [46–50]. The authors of each of these studies termed these two factors innovative and adaptive, patterned after Kirton’s [51] Adaptor-Innovator theory (KAI). The KAI postulates problem-solving and creativity are often manifested in one of two styles; the adaptive style is characterized by doing things better, while the innovator style is characterized by doing things differently [51]. In each of these studies fluency and originality loaded onto the innovative factor, and elaboration and abstractness of titles loaded onto the adaptive factor [46–50]. Resistance to premature closure had less consistent results, loading onto both the innovative and adaptive factors in [46] and [49], just the innovative factor in [48], and just the adaptive factor in Humble et al. [47] and [50].

If the KAI theory is correct, it would present a problem for TTCT because the tests’ subscores and global score do not align with the theory. Instead of two subscores—innovation and adaptation—each version of the TTCT produces a global score and three (for the TTCT-V) or six (for the TTCT-F) subscores. This may represent a major deficit in the internal validity evidence of the TTCT and its ability to give test users a correct understanding of examinees’ divergent thinking. Because our study is psychometric in nature, it will not provide definite answers to the question of how creativity and intelligence relate to one another as constructs. Instead, this study is limited to investigating the degree to which divergent thinking and intelligence subtests form separate latent factors.

In summary, one of the intransigent problems with the TTCT is that its factor structure (1) may not align with leading theories of the nature of creativity, and (2) the interpretation of its scores—especially the global scores that the TTCT produces. This means that the construct validity and internal validity of the TTCT-F and TTCT-V are disputed. An investigation into the factor structure of the tests could clarify the psychometric properties of both versions of the TTCT.

Factor structure of intelligence tests

In contrast with the TTCT, the factor structure of scores from intelligence tests is well established. Almost all cognitive test batteries produce a series of scores that either load onto a single factor or produce a number of factors that, in turn, load onto a single general factor [65, 66]. This general factor has been called g since Spearman [67] discovered it over a century ago. Among psychologists studying intelligence, g is a mainstream theory, and there is strong evidence that every cognitive task loads on g to some extent [64]. Spearman [68, pp. 197–198] called this phenomenon the “indifference of the indicator,” and it has led some experts to argue that every task in life that requires cognitive work is its own intelligence test, which would explain why IQ scores correlate with so many life outcomes [69–71].

The indifference of the indicator presents a major challenge to the widely accepted belief that creativity tests—especially the TTCT with its focus on divergent thinking—measure a distinct construct from intelligence. TTCT test content is unquestionably cognitive, and according to the theory of the indifference of the indicator, TTCT content should measure g, at least partially. Moreover, in the past, some researchers designing tests to measure other constructs have accidentally created tests that measured intelligence [64, Chapter 7]. For example, Sanders et al. [72] found that the Defining Issues Test, a test designed to measure moral development and reasoning, is g-loaded and is a moderately good measure of verbal intelligence. Likewise, literacy tests are highly g loaded and—in American samples—do not seem to measure a construct that is distinct from intelligence [73]. These examples and the strong evidence in favor of the indifference of the indicator raise the possibility that the TTCT is actually a measure of intelligence. The best way to investigate this possibility is through psychometric study of the factor structure of scores from the TTCT and an intelligence test when given to the same sample of examinees.

The factor structures of both the TTCT and intelligence tests have been studied independently, but we have been unable to identify any research examining the factor structure of creativity and intelligence tests at the same time to determine whether these tests measure the same latent construct, or multiple constructs (and how multiple constructs might be related to one another). Our goal in conducting this study is to determine the factor structure of subtests drawn from intelligence and divergent thinking tests when administered together. Through this study we aimed to determine the degree to which divergent thinking subtests and intelligence subtests measures separate constructs. We used confirmatory factor analysis to determine which of a series of plausible factor structures provides the best fit for the subtest data.

Theoretically plausible factor structures

Sternberg and O’Hara [74] described five possible ways that could describe the relationship between intelligence and creativity:

• Creativity is a component of intelligence.
• Intelligence is a component of creativity.
• Creativity and intelligence are different constructs with overlapping components and/or mental processes.
• Creativity and intelligence are different labels for what are, substantially, the same problem-solving construct.
• Creativity and intelligence are separate constructs, with any correlations between the two being incidental.

Sternberg and O’Hara [74] found that there was evidence supporting all of these viewpoints, making it hard to distinguish which one is the best description of the relationship between intelligence and creativity. Years later, Karwowski et al. [6] added another possibility: that intelligence is a necessary but not sufficient condition for high creativity, which may explain the positive correlations between the two but also why many high-IQ individuals fail to engage in creative behaviors. Complicating the picture is that the relationship between the two constructs may be dependent on the context, [75] with a degree of domain knowledge often being necessary for a person to generate creative ideas [2].

With the exact nature of the relationship between intelligence and creativity remaining an unresolved question, we posited that the positive correlation between the two constructs may be partially due to a measurement artifact that some tasks that measure creativity may be—like all other cognitive tasks—partially measuring intelligence. To investigate this possibility, we conducted this study to determine whether the divergent thinking tasks on the TTCT are measures of general intelligence (i.e., are g-loaded). Such a study would help psychologists in both fields (i.e., creativity and intelligence) determine whether the apparent relationship between scores on tests measuring intelligence and creativity is due to a measurement artifact or to a real relationship between the two constructs. Understanding the nature of the correlation between intelligence and creativity would help clarify (and possibly support or falsify) theories of the relationships between the two constructs and improve research in future studies.

The possibility that divergent thinking tests also measure intelligence is not far-fetched, and the history of intelligence testing has several examples of measures of divergent thinking being interpreted as measuring intelligence. Over 100 years ago, Binet and Simon [76, pp. 229–230] included a task on their second intelligence test in which a child examinee was asked to name as many words as possible in three minutes. Although the task was scored quantitively by tallying up the total number of words that the examinee produced, Binet discussed the qualitative differences in responses among children, noting the different categories of words that some examinees generated or the uniqueness of their responses. Modern creativity researchers would recognize Binet’s quantitative scoring procedure as a measure of fluency and his qualitative analysis as touching upon originality and flexibility in responses [43]. More recently, Carroll [66] identified fluency as a manifestation of a broad mental retrieval ability that was subsumed by a general intelligence factor. Other researchers from the psychometric tradition have also suggested that common measures of creativity and/or divergent thinking could be measuring aspects of intelligence [13].

From a psychometric perspective, there are several factor structures that we believe are plausible for explaining the relationship between scores on divergent thinking and intelligence tests: (a) distinct but correlated constructs, (b) scores loading directly on a g factor, and (c) a hierarchical structure with g as a second-order factor. We will briefly describe each of these possibilities here and then describe exact models we will use in our study in the Methods section.

Possibility (a), which is a factor structure of correlated separate constructs, would emerge if intelligence and divergent thinking subtests produce separate factors. This is the factor structure suggested by divergent validity research showing that intelligence and divergent thinking tests do not strongly correlate with one another (e.g., [11]). In the most straightforward form, this could occur if the TTCT and intelligence tests each produced their own factor(s) which were, in turn, correlated with one another. Structure (b) would occur if subscores from the divergent thinking subtests and intelligence tests all formed a single, undifferentiated general factor. This would support the intelligence community’s belief that all cognitive tasks measure g to some extent and be strong evidence that Spearman’s [68] theory of the indifference of the indicator is correct. Finally, possibility (c) would be a hybrid model of (a) and (b) where a number of factors can combine to form a single second-order g factor. This type of structure would support intelligence theorists’ belief in the universality of g but also permit the existence of first-order factors—such as divergent thinking and intelligence factors that span a number of subscores. In this study, we assessed 1–4 measurement models of each type in order to better understand how intelligence test scores and divergent thinking test scores are related.

Ethics

The study received ethical approval from the Utah Valley University Institutional Review Board, protocol #441. Informed consent was obtained at two time points: when examinees took the cognitive ability test online and again at the beginning of the in-person testing session (when the TTCT tests were administered). The initial informed consent was recorded online, while the informed consent obtained at the beginning of the in-person testing session was obtained through a signed written consent document.

Instruments

We collected a total of thirteen subscores drawn from three professionally developed psychometric tests: the Torrance Tests of Creative Thinking Figural Test A, the Torrance Tests of Creative Thinking Verbal Test A, and an abbreviated version of the International Cognitive Ability Resource (ICAR) test.

Data and scoring

For both TCTT tests, we used professional scorers (employed by the test publisher) to score the tests in order to improve data quality and reduce scoring errors and subjectivity that novice scorers are susceptible to. These scorers followed procedures in the two TTCT scoring manuals. [77] In short, higher scores on fluency subscales on both tests correspond to a larger number of valid responses. Higher originality subscores on both tests correspond to more responses that were unusual in the test’s norm sample. For the TTCT-V, higher flexibility subscores indicate a larger number of categories (pre-defined in the test manual) that responses could be placed into, and higher creative strengths checklist subscores indicate a larger number of special types of responses (e.g., indicating emotion, humor, fantasy) and/or more frequent examples of these responses. For the TTCT-F, higher elaboration subscores are indicative of more detailed drawings, while higher scores on the abstract titles subscale correspond to a larger number of titles that are not literal and/or achieve higher levels of abstractness. Finally, the resistance to closure subscore is higher for examinees who do not complete stimulus figures to form small, enclosed polygons or shapes.

In analysis, we used TTCT subscores that had not been converted to standardized scores or age or grade percentiles. For the ICAR test, the raw scores were the number of test items correctly answered on each subtest. In accordance with our published pre-registration protocol, [80] data from examinees who left the testing setting early and did not attempt all three tests were not used in the analyses.

The scoring system for the TTCT Verbal and Figural tests is based on counts of valid responses (fluency), which are then used to produce scores for unusual responses (originality) and the number of categories responses fit into (flexibility). The TTCT Figural also uses counts to produce scores for abstract titles, resistance to closure, and the creative strengths checklist. Because achieving a high subscore on originality, elaboration, and any of the other scores is dependent on first generating a number of responses, these scores are confounded with the test’s fluency scores. Over the years, researchers have proposed different methods for handling this confounding, [43, 81] and there is no one clear correct method for doing so. The method we have chosen and pre-registered is to divide each task’s subscore by the fluency subscore for the same task to produce a rate per valid response (e.g., a rate of responses that qualify as original). We then summed the subscores that measure the same aspect of divergent thinking, and used these summed rates in our analyses to create adjusted scores. These altered scores can be interpreted as a respondent’s average rate of divergent responses (i.e., original responses, elaborate responses, etc.) per valid response. Note that the creative strengths checklist was not adjusted in this fashion because its score is derived across all tasks on the TTCT-F, and that it is not possible to control for fluency at the task level. Therefore, this score is still partially confounded by fluency.

Sample

Sample size was set by funding constraints but was large enough to meet guidelines for conducting confirmatory factor analysis studies. Our funding permitted the purchase of test materials for 460 examinees. The goal was to have sufficient data from at least 399 examinees after data loss due to incomplete tests, missing subscores, or other problems. This would allow every model to have at least 20 sample members per estimated path. This goal exceeded the recommended minimum sample size for convergence and parameter estimate stability in confirmatory factor analysis [82]. Statistical power for the chi-squared difference tests could not be calculated because realistic parameter estimates require plausible a priori estimates of all other model parameters [83], but because this is the first confirmatory factor analysis of both intelligence and creativity subscores, these estimates are not available.

Of the 519 participants who took the ICAR, 427 later took the TTCT and had analyzable data. Fig 1 shows the number of people who left the study at each stage and/or the reasons data were unanalyzable. By far, the most common reason for subject loss was that participants who completed the ICAR online did not show up at a TTCT testing session at a later date (n = 82). One participant participated in a TTCT data collection session but did not complete it. Of those who took the TTCT and ICAR, two participants took the TTCT tests twice, and data from one of their two sessions were randomly selected to be merged with their ICAR data. (The data from the other testing session were dropped from the study.) Two other participants’ ICAR and TTCT data could not be merged because names and/or student ID numbers on the ICAR could not be matched to corresponding information on the TTCT tests. These two individuals were eliminated from the study. Finally, five participants had unusable adjusted scores because they had a zero fluency score on one more TTCT-F or TTCT-V task, which resulted in an undefined value for their adjusted score. Their data were also not analyzed in this study.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Subject flow diagram for the study.

https://doi.org/10.1371/journal.pone.0274921.g001

The sample consisted of a convenience sample of students attending a large open-enrollment university in the western United States and members of the surrounding community. Students received course credit for participating in the testing session or for inviting a community member who participated. Psychology students at this university are required to participate in research as part of their education. Examinees took the ICAR online (its typical administration format) up to three weeks before their in-person testing session. Total testing lasted 120–150 minutes (including breaks between tests), with the TTCT tests being administered in person in a session that lasted up to 110 minutes. TTCT test order was randomized for each session.

Analysis

We calculated basic descriptive statistics for all variables: means and standard deviations for all subtest scores and demographic variables that are interval- or ratio-level data, along with frequency tables for nominal- and ordinal-level demographic variables. We also created a correlation table for all subtest scores.

Our plan was to use confirmatory factor analysis to examine nine plausible models for how the subscores on the TTCT tests and the ICAR test could relate to one another. These models are summarized briefly in Table 1. All confirmatory factor analyses were performed with MPlus 8.4. We believe that a Registered Report fits our study design and analysis plan because “Registered Reports can be an empowering venue for testing new theories or arbitrating between competing theories” [84, p. 570]. Having locked-in and pre-registered models removed any subjectivity from our study and ensured that the model selection process occurred without regard for our preference for any particular model.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Proposed models for interrelationships among TTCT and ICAR subscores.

https://doi.org/10.1371/journal.pone.0274921.t001

The nine models fall into the three groups described in the introduction section and are all either hierarchical or non-hierarchical models. Models 1a, 2a, 3a, and 4a all represent a multifactor relationship between creativity and intelligence subscores, where all factors are intercorrelated. Model 5 is a model where all scores load directly onto a general factor. The hybrid models are Models 1b, 2b, 3b, and 4b, which all have 2–4 initial factors which then load on a general factor, making them hierarchical models. All other models (i.e., 1a, 2a, 3a, 4a, and 5) are non-hierarchical models. Almost all non-hierarchical models have a corresponding hierarchical model with the same number of factors, but with the addition of a general second-order g factor. The exception to this is Model 5, which is a congeneric model where all subscores load directly onto a general factor.

Models 1a and 1b have three first-order factors that are based on the tests, with both forms of the TTCT and the ICAR each forming its own factor based on its subscores. Models 2a and 2b are similar, but form two first-order factors: one for all TTCT subscores and another for ICAR subscores. These models would be appropriate if both forms of the TTCT measure divergent thinking and the ICAR measures a separate cognitive ability (i.e., intelligence). Models 3a and 3b have two second-order factors based on the subtest stimuli, with verbal scores all forming a factor and non-verbal scores forming a separate, correlated factor. These models would support the traditional dichotomization between these types of stimuli on intelligence tests, which dates back to David Wechsler. Models 4a and 4b are based on empirical evidence that subscores on the TTCT Figural test do not fit a one-factor model [48, 50] and instead are best represented with a two-factor model comprising of an “innovative factor” and an “adaptive” factor, which would support Kirton’s [51] adaptor-innovator theory. (Another measure of divergent thinking showed a similar two-factor structure; see [13]). All nine models are diagrammed in Figs 2–10. There were no attempts to modify models in order to improve fit. However, because prior research shows that the creative strengths subscore on the TTCT Figural test can lead to poor model fit, [48] we also tested these models without the creative strengths subscore. These results are relegated to a supplemental file because the TTCT’s creator saw the creative strengths subscore as being essential for understanding a person’s performance on the TTCT Figural test [50].

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Confirmatory factor analysis diagram for model 1a.

https://doi.org/10.1371/journal.pone.0274921.g002

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Confirmatory factor analysis diagram for model 2a.

https://doi.org/10.1371/journal.pone.0274921.g003

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. Confirmatory factor analysis diagram for model 3a.

https://doi.org/10.1371/journal.pone.0274921.g004

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. Confirmatory factor analysis diagram for model 4a.

https://doi.org/10.1371/journal.pone.0274921.g005

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 6. Confirmatory factor analysis diagram for model 5.

https://doi.org/10.1371/journal.pone.0274921.g006

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 7. Confirmatory factor analysis diagram for model 1b.

https://doi.org/10.1371/journal.pone.0274921.g007

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 8. Confirmatory factor analysis diagram for model 2b.

https://doi.org/10.1371/journal.pone.0274921.g008

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 9. Confirmatory factor analysis diagram for model 3b.

https://doi.org/10.1371/journal.pone.0274921.g009

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 10. Confirmatory factor analysis diagram for model 4b.

https://doi.org/10.1371/journal.pone.0274921.g010

Models were identified with the reference variable strategy, where one variable’s factor loading is set to 1.0. The fluency scores and the matrix reasoning score were used for this purpose because these scores tend to have the strongest loadings on TTCT and intelligence factors in exploratory factor analyses [85, 86]. For hierarchical models, the factor loading for the first-order ICAR factor on the second-order g factor was set to 1.0 because if there is a general construct that parsimoniously explains performance on the subtests, then it is likely a lower-level general intelligence factor should have a strong loading from a first-order intelligence factor.

Interpretation

Fit statistics were used to evaluate model fit and compare models with one another. We used the chi-squared value, the comparative fit index (CFI), Tucker-Lewis Index (TLI), root mean square error of approximation (RMSEA) with 90% confidence interval, standardized root mean square residual (SRMR), Akaike information criterion (AIC), and Bayesian information criterion (BIC). These fit indices are suitable for making comparisons among competing models, and they are a cross-section of fit statistics with compensatory strengths and weaknesses [87, 88]. For this study, models were judged to have acceptable fit if they have an SRMR value ≤ .08 and at least one of the following: (1) CFI ≥ .90, (2) TLI ≥ .90, or (3) RMSEA ≤ .08. These statistics were used to judge the best fitting model(s) among the nine. Models were favored when their SRMR and RMSEA values are closer to zero and their CFI and TLI values are closer to 1.0.

The AIC and BIC were used to compare all models to one another, with lower values of these statistics indicating better model fit. When comparing models with the same degrees of freedom, both fit statistics will favor the same model, and the model with the lowest AIC and BIC will be preferred. When models have differing degrees of freedom, the penalty for a more complex model (i.e., with fewer degrees of freedom) will be more severe for the BIC than the AIC. Therefore, if a more complex model has a lower BIC than a simpler model, then it would be preferred because this would indicate that the model fits the data much better than the simpler model, despite the loss of parsimony.

Chi-squared difference tests were used to compare well-fitting nested models. Table 1 indicates which models are nested within one another. Model 5 is the most general of these models, with Models 1a, 2a, 3a, and 4a nested within it. These latter models were to be compared to Model 5 in a sequential fashion by constraining a correlation between the relevant first-order factors to 1.0—in order to force these factors to merge—and conducting a chi-squared difference test with k—1 degree of freedom, where k is the number of factors being merged. There are two sequences of nested models that can be tested in this fashion. The first starts with the complex Model 4a, and testing whether it is statistically equal to Model 1a, followed by a test to determine whether Model 2a is statistically equal to Model 5. The second group of nested models tested whether Model 3a is statistically equal to Model 5 in the same fashion. All of these comparisons were planned, on the condition that multiple nested models fit the data well (based on the fit statistics). Any tests that produce a non-statistically significant result (with p > .05) would indicate that the two nested models are equivalent and that the simpler model with more degrees of freedom should be preferred. Note that to test nested models, we constrained correlations between factors in the more complex model to be equal to 1.0 to create the more parsimonious model and test whether this model is statistically different from the more complex model. Because a constrained correlation of r = 1.0 creates a boundary constraint (because the correlation between factors cannot be greater than 1.0), we planned to use the appropriate mixed chi-squared distribution for each comparison (see [89] for details).

After identifying the best model(s) that fit the data, we interpreted the results in light of the different plausible relationships that divergent thinking test scores and intelligence test scores could have. If Models 1a or 2a fit best, that would indicate that the TTCT and ICAR measure separate but correlated constructs. If Model 3a fit best, then we would interpret this to indicate that the TTCT and ICAR measure a mix of verbal and non-verbal reasoning behaviors, whereas Model 4a being the best fitting model would support the view that TTCT tests measure different constructs and that the TTCT Figural test has a multifactorial structure. A best fitting Model 5 would indicate that all TTCT and ICAR tests are direct measures of g and that there are no separate constructs that these tests measure. If Models 1b or 2b fit the data best, then we would interpret this as indicating that the TTCT and ICAR measure separate first-order factors but that these then combine to form a second-order g factor. If Model 3b has the best fit indices, then this would mean that the tests measure a mix of verbal and non-verbal behaviors, which then coalesce into a g factor. Finally, if Model 4b fits best, then this would indicate that there is a general g factor among the subscores in the study, but that these subscores also form mediating factors for intelligence, verbal divergent thinking, an adaptive figural behavior, and innovative figural responses.

There was the possibility that no model will have adequate fit. If this occurs, we planned to interpret this result to indicate that the factor structure of divergent thinking and intelligence subtest scores does not conform to any a priori theorized structure and tests and fit statistic comparisons to find the best model are unneeded. The factor structure of these two tests in relation to each other would remain unresolved. In our registered report protocol, we intended not to engage in any exploratory analyses (e.g., exploratory factor analysis, modification of confirmatory models to improve fit) if no pre-specified confirmatory models were found to fit the data. As indicated later in this article, we later did conduct an exploratory analysis after obtaining approval from the article’s editor.

Predictions

We had no firm predictions about what this study may show, and we were agnostic about the results. To us, all nine of these models were plausible, and we did not have a strong belief about which may be the best fitting model. We also recognized that because fit statistics are sensitive to different aspects of the data and of model misspecification that we could get contradictory indications of what “the best fitting model” is. For example, a model could have the lowest BIC but higher SRMR and RMSEA values than other models. If contradictory results appear in the fit statistics, we planned to report this and interpret it to indicate that we could not fully resolve the question of how divergent thinking test scores and intelligence subtest scores interrelate but that we had narrowed the range of possibilities. There were no attempts to modify models in order to improve fit.

Open science practices

This project is designed to conform to all standards of open science. All methods and materials (except those protected by copyright or confidentiality guidelines) are available to view at https://osf.io/8rpfz/. Publication as a Registered Report Protocol [80] functioned as a pre-registration for the study. To supplement final publication of the study, we have made all raw data, analysis syntax, and output files freely available online at https://osf.io/8rpfz. We have also made a pre-print of the results available when submitting to the journal. Additionally, we will provide the response forms available to researchers who explain their qualifications and interests to us, so that they can re-analyze our data with alternate scoring methods (such as those explained by Reiter-Palmon et al. [43]). Readers should note that making response booklets and examinee responses openly available online would violate copyright on the TTCT and item confidentiality of both tests, so we will control access to these materials.

Descriptive statistics

Table 2 lists the means, standard deviations, and the reliability coefficients (i.e., Cronbach’s α values) on all analyzed variables. Compared to the averages for the norm sample, the mean raw subscores of our sample were higher than average for the TTCT-V (d = .18 to .50) and lower than average for the TTCT-F subscores (d = -.74 to -.05). These effect sizes were calculated from raw scores that had been converted to standard scores that have a mean of 100 and standard deviation of 20 for each subscore. A summary report of mean standard scores for the sample is available on the study’s OSF page at https://osf.io/8rpfz/. Table 3 is a correlation table of the same variables for our sample members. For both of these tables, n = 427 sample members who had complete data on all variables in the table.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Descriptive statistics of variables used in the CFA models.

https://doi.org/10.1371/journal.pone.0274921.t002

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Correlation table for variables used in the CFA models.

https://doi.org/10.1371/journal.pone.0274921.t003

Table 4 displays the fit statistics for Models 1–5. Among the six models which had converging results, Models 1a, 1b, and 4a had the best values for their fit indices. However, all of these models had specification problems. Models 1a and 1b had a negative residual variance for the TTCT-F adjusted elaboration score. Model 1b also had the g factor defined by higher-order loadings that were all statistically equal to zero (all two-tailed ps ≥ .939). For Model 4a, the factor loadings (and correlation coefficients with other latent factors) for the TTCT-F Adaptive factor were undefined.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 4. Fit statistics for CFA models.

https://doi.org/10.1371/journal.pone.0274921.t004

Moreover, none of the models met the pre-registered standards for acceptable model fit (i.e., an SRMR value ≤ .08 and at least one of the following: (1) CFI ≥ .90, (2) TLI ≥ .90, or (3) RMSEA ≤ .08). By these standards, none of the models can be defined as representing the data well. The S1 File reports results of the same models with the creative strengths checklist score. These models also did not reach the standards for sufficient model fit.

Limitations

One advantage of our study is also a limitation: as authors of a registered report, we were locked into a scoring system and analysis plan. This strengthened our confidence in our findings because we knew they could not be the product of questionable research practices or unconscious adjustments to our study as we analyzed the data. However, some of the analysis decisions are still arbitrary, such as the method used to adjust scores for the confounding influence of fluency (which greatly reduced the magnitude of correlations among TTCT variables and flipped many from positive to negative correlations). Because there are other scoring alterations which can control for fluency, we have made our altered scores and the original scores available in our open data set at https://osf.io/8rpfz/ so that researchers can use other methods for controlling for the confound of fluency and test our models (or other hypotheses) accordingly. Alternate methods of controlling for fluency may be useful for identifying causes of model misfit in our study and improve the interpretation of TTCT scores.

Prior studies using raw scores—instead of scores adjusted for the influence of fluency—have shown that TTCT-F data can fit confirmatory factor analysis models [47, 48]. The differing results in our study may indicate that a well-fitting measurement model with raw TTCT scores may be an artifact of the confounding influence of the fluency score on the other scores. If this is true, then it raises fundamental questions about the validity of interpretations of originality, flexibility, and other non-fluency TTCT scores.

This study is also limited by the convenience sample. Most participants were college students, though the open enrollment nature of the university makes restriction of range much less of a problem for the intelligence test scores than at many universities. Yet, any restriction of range did not prevent the ICAR data from having an extremely good model fit by itself. We have no reason to suspect that the convenience sample suffered from restriction of range in the creativity scores. Score summaries (available at https://osf.io/8rpfz/) show that standard scores spanned a very wide range of scores and that there were no apparent ceiling or floor effects. Yet, the degree of generalizability of these findings is still unclear.

Finally, it is important to recognize that there is some subjectivity inherent with scoring some of the tasks on the TTCT. For example, the TTCT-F’s creative strengths checklist requires scorers to make judgements about the humor, richness, expressiveness of titles, and other aspects of responses. Even core aspects of the TTCT-F and TTCT-V, such as originality, can require subjective judgment in the scoring process, especially when examinees’ responses are somewhat ambiguous. We believe that using professional scorers employed by the testing company—who have extensive training and experience with the TTCT—would minimize subjectivity in the scoring process and increase the consistency of subjective judgements. It is also important to note that subjectivity in scoring is not a unique problem with the TTCT and that other measures of creativity and divergent thinking have the same problem. Researchers cannot eliminate subjectivity in scoring creativity tests; the best option is to manage and minimize it, and we have tried to do so as much as possible.