Study 1: Extinct English words as nonwords

Much like an extinct species or an extinct volcano, words that are no longer commonly used in a language might be described as “extinct words” (the equally descriptive phrase “linguistic fossils” is used in [19] pg. 9). An example of such a word is upknocking, which describes a 19th century occupation (i.e., waking people at a pre-determined time in the early morning) that is no longer a form of employment given the widespread availability of alarm clocks [19]. Because such words are not presently used in everyday speech/text, the definitions associated with these sequences of letters are not likely to be known by most human speakers of contemporary English, and can therefore be considered nonwords.

In this study words that have fallen out of common usage in contemporary (American) English were selected from the collection of words found in [19], and presented to ChatGPT with the prompt: Define <word > . The number of words not defined (e.g., Chat GPT responded that the word “…is not a recognized term in standard English dictionaries…”), and the number of words defined correctly (i.e., the definition matched the definition in [19]) were assessed. The content of definitions that were provided for some words was examined further. In these cases, Chat GPT provided a definition, but the definition did not match the definition in [19]. Google Books Ngram Viewer [20] was also used to determine the year of peak usage for each extinct word.

Method

A sample of words was selected from a dictionary of extinct words [19]. The words included in [19] do not constitute an exhaustive list of such words, but were subjectively selected due to their “eccentric phonic essence” (i.e., they sounded funny), and because they captured “…humorous aspects of Old World life” ( [19]; pg. 9). Further, technical terms and non-English foreignisms were excluded from the collection [19].

From the collection of words in [19], 52 items were selected with 2 words starting with each of the 26 letters of the English alphabet. Phrases and hyphenated or monosyllabic words were excluded. Preference was given to words that contained prefixes, suffixes, and base words (i.e., they contained some (pseudo)morphological complexity), but not words that contained transparent morphology (e.g., even if one has never heard the word biblioklept it might be easily inferred that it refers to someone who steals books), or extinct words that were related to a word that remained in common usage (e.g., peccable/impeccable or ruly/unruly). Words that resembled a typographical or spelling error were excluded (e.g., bridlegged). When possible, phonotactic and orthographic regularity was preferred, and one word in each letter pair had a singleton in the onset and a consonant cluster in the onset. See the Supporting Information for the words that were selected from [19]. From June 3–7, 2024, varying numbers of the 52 words were pseudo-randomly selected and presented to Chat GPT 4o with the prompt: Define <word > .

Results

Of the 52 extinct words that were presented, Chat GPT provided a correct definition (i.e., matching the definition in [19], or the definition from [21]) for 36 items (69.2%). For 11 items (21.2%), Chat GPT was not able to provide a definition, and instead responded with:

[cue word] does not appear to be a recognized word in standard English dictionaries or common usage. It might be a misspelling, a very obscure term, or specific to a certain dialect or context that is not widely documented. If you have more context or details about where you encountered the term, I could help you further in trying to determine its meaning.

For the remaining 5 items (9.6%), Chat GPT provided a definition, but that definition did not match the definition in [19 or 21]. Inspection of the definitions that fell into this category shows that 3 of the definitions were connected to “foreign” words or concepts. For example, wangary (soft and flabby meat; [19]) is (correctly) identified by Chat GPT as a town in South Australia. The word lagam (goods which are sunk with a buoy attached to facilitate later recovery; [19]) is identified by Chat GPT as a form of the Tagalog verb “lagom” (meaning to summarize). The word zoldering (an opprobrious epithet; [19]) is identified by Chat GPT as being similar to the Dutch word “zolder” (“attic” in English), and therefore could relate to constructing or renovating an attic, or be a regional term for the attic space itself.

For the remaining 2 items, Chat GPT provided a definition, but that definition did not match the definition in [19 or 21]. The word flothery (slovenly, but attempting to be fine and showy; [19]) was defined by Chat GPT as “…light, airy, or fluffy…soft and delicate in texture or appearance, akin to fluff or froth.” The word tantrels (idle, unemployed people; [19]) was defined by Chat GPT as “…youngsters who are playful in a troublesome or disorderly manner…” These definitions might be described as a form of hallucination, or output from an LLM that is nonsensical or inaccurate [22,23].

Google Books Ngram Viewer (July 2024 dataset; [20]), which contains words and phrases from books published from 1800–2022, was used to determine the year in which the 52 words used in the present study experienced peak frequency of usage. Data were available for 40 of the 52 words. An ANOVA (using [24]) revealed that there was no difference in the year of peak frequency of usage (F (2,37) = 0.42, p = .66) for the words that were defined correctly (n = 30, mean = 1855.33 CE, sd = 47.66), the words that Chat GPT could not define (n = 8, mean = 1868.5 CE, sd = 52.80), and the words that Chat GPT defined incorrectly (n = 2, mean = 1879.00 CE, sd = 2.83).

Discussion

In the present study extinct words were presented to Chat GPT with the prompt to define the word. Despite the words not being frequently used in English for approximately 165 years, Chat GPT produced a correct definition to 69.2% of the words. Given the amount of time since these letter sequences were frequently used to convey meaning in English (and the inclusion of such words in a curated collection of extinct words; [19]), it is unlikely that a human speaker of contemporary English would attain a level of accuracy comparable to Chat GPT if asked to define the same extinct words. Rather, a speaker of contemporary English would most likely consider all of these letter sequences to be devoid of meaning, and therefore nonwords, which was indicated for 21.2% of the words by the response from Chat GPT that the word “…does not appear to be a recognized word in standard English dictionaries or common usage.” For the remaining 9.6% of the words, Chat GPT provided definitions that were “hallucinations.” That is, the definitions were not consistent with the definitions found in established sources, and were instead nonsensical or inaccurate [22,23].

The results of the present study demonstrate one way that artificial intelligence can be used to augment or assist in some way human intelligence. Consider the human cognitive process of collective memory, or the body of knowledge that we know (or think we know) about the past [25]. Previous studies have shown that collective memory for US Presidents is limited to the 9–10 presidents that served most recently in one’s lifetime [26]. The 9–10 presidents recalled most accurately by students tested in 1974 differed from the 9–10 presidents recalled most accurately by students tested in 1991 (and the same for students tested in 2009 [26]). Beyond that number of presidents, recall performance was significantly reduced in all students, regardless of when they were tested (see [26] for additional caveats).

If we consider “words in the English language” to be a body of knowledge that we (think we) know, then we should expect that over time some words from that body of knowledge will be lost from our collective memory (indeed, enough of them to fill a book; [19]) just as certain US presidents were lost from collective memory in the study by [26]. A well-designed/well-trained AI could enable a modern-day English speaker to reach beyond the limits of their collective lexical memory to correctly identify a letter sequence as a meaningful English word (even if it is a word that is no longer in common use), as Chat GPT did with 69.2% of the extinct words in the present study. Such a cognitive tool could also make a modern-day English speaker aware of other changes to the language that may have taken generations to occur [27], further extending knowledge about the past for a modern-day English speaker.

Study 2: Words from another language as nonwords

Whereas Study 1 examined meanings (i.e., semantic information) that might be associated with letter sequences, the present study examined phonological information, or knowledge of how the letters in a written word sound when spoken aloud. Given that Large Language Models like Chat GPT receive written/text input during training, any knowledge of how a word sounds when spoken aloud must be obtained indirectly. Across languages there is variability in the depth of the orthography, or the consistency with which letters or sequences of letters map to the phonemes in that language (i.e., [17]). However, within a language, even for a language with a relatively deep orthography like English, there are some regularities and patterns (as well as exceptions) that might be observed and potentially exploited [28]. The present study used techniques from machine psycholinguistics to examine how Chat GPT might use the phonological information that may have been extracted from the texts that were used as input during training.

Previous studies of LLMs used several language-related tasks—grapheme-to-phoneme conversion, syllable counting, and generating a word that rhymes with the cue—commonly used to assess engineered systems that perform natural language processing, text-to-speech synthesis, and automatic speech recognition systems [29]. The same tasks have also been shown to predict success in learning to read in human children, and have been used by speech-language pathologists and other professionals to screen for potential language or reading disorders [30]. In the work by [29], these tasks were used to assess the performance of six different LLMs. It was found that no single model excelled at all three tasks. Rather, one model might have done well in one task, but quite poorly in another. Further, in the grapheme-to-phoneme conversion task, several of the models performed at a level that approached human performance in the task. However, in the syllable counting and rhyme generation tasks, all of the models lagged significantly behind the level of performance observed in humans.

In the present study the LLM was prompted to respond with a word that sounded like the nonword stimulus. A similar task has been used with human speakers of English using real words [15] or English-like nonwords [14] as the prompt. In the present study the nonwords were actually words from another language (i.e., Spanish). However, the same metric used in previous studies with humans [14–15] was used in the present study to determine how similar the response generated by Chat GPT sounded to the cue item.

Method

Twenty bi-syllabic Spanish words (10 had masculine gender, ending in -o; 10 had feminine gender, ending in -a) were selected from a list of words previously used in [31]. Spanish words with 2 syllables were selected because an analysis reported in [32] found that 90% of Spanish words had two or three syllables, and 80% of English words had one or two syllables (see also [33]). Thus, two syllable words represented a region of overlap for the two languages, increasing the likelihood that the Spanish words would still be somewhat word-like in English (even though the letter sequences are devoid of meaning in English).

Although the real words in Spanish were considered nonwords in English, each Spanish word was phonologically similar to at least one English word as determined by computational analysis [16]. Phonological similarity of the Spanish and English words was defined as the phonological transcription of the Spanish words differing by a single phoneme (via addition, deletion or substitution) from the phonological transcription of a real word in English. From 13–15 August 2024 the Spanish words were pseudo-randomly presented to Chat GPT 4o, first with the prompt Give me a word that sounds like <word > , and, after all 20 items had been presented, a second time (in the same order) with the prompt Give me an English word that sounds like <word > . The second round of stimulus presentation and the addition of the word “English” in the prompt occurred because after 3 trials Chat GPT 4o responded to the first prompt with Spanish rather than English words.

Results

The responses provided by Chat GPT to the nonwords/Spanish words used as stimuli in Study 2 are presented in Table 1. The nonwords are listed in the table in the pseudo-random order in which they were initially presented. English translations are provided in parentheses.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Responses from Chat GPT from the nonword stimuli in Study 2.

https://doi.org/10.1371/journal.pone.0325612.t001

The same order of presentation was used when presenting the nonwords with the second prompt. Note that after 3 trials using the first prompt (Give me a word that sounds like <word>) Chat GPT produced a Spanish word as a response. The one exception during the first round of presentation was the response rubo to the stimulus item tubo (pipe). Although rubo is not a word used in contemporary Spanish, it is a word [I steal] in Italian, another Romance language, and in Galician, another language spoken in Spain ([steal]). Further, rubo is similar to the Spanish words rubor (blush) and rubí (ruby).

In the presentation of the second prompt (Give me an English word that sounds like <word>), Chat GPT responded with real English words on most trials. The exceptions included 2 responses that might be described as short phrases (in response to mito and nata), and 2 instances in which the stimulus item was simply repeated in the response (dato and gato). In the case of gato, additional information was included in the response:

An English word that sounds like “gato” is “gato” (in the context of the Spanish word used in English conversation). A close match would be “gato” when pronounced similarly to “gateau” (a French word for cake).

The same metric used in previous studies with humans [14–15] was used to determine “how similar” the response generated by Chat GPT sounded to the cue item (i.e., how many phonemes differed between the cue and response). For the responses to the first prompt 85% (17 of 20) differed from the stimulus word by a single phoneme, and 15% (3 of 20) differed from the stimulus word by more than 1 phoneme. The status of the response (e.g., English word, Spanish word, nonword, etc.) was not considered in this assessment. Further, the type of phoneme change (e.g., addition, deletion, or substitution) was not considered in this assessment.

For the responses to the second prompt, 20% (4 of 20) of the responses were phonologically identical to the stimulus word (i.e., 0 phonemes differed), either because the stimulus word was simply repeated (in the case of gato and dato), or because the English “phrase” that was provided as a response was phonologically identical to the Spanish word/nonword stimulus (in the case of mito/“meat-o” and nata/ “not a”). Further, 45% (9 of 20) of the responses differed by a single phoneme, and 35% (7 of 20) differed from the stimulus word by more than 1 phoneme. Again, the status of the response (e.g., English word, Spanish word, nonword, phrase, etc.) was not considered in this assessment, nor was the type of phoneme change (e.g., addition, deletion, or substitution).

Discussion

In Study 2 Chat GPT was prompted to provide a word that sounded similar to a sequence of letters provided as input. Although the letter sequences presented as the stimulus were real words in Spanish, the letter/phoneme sequences were devoid of meaning in English, making them English-like nonwords.

In the first part of this study, the prompt “Give me a word that sounds like <word>” was used. For the first 3 trails, Chat GPT provided an English word as a response to the nonword, but in the subsequent trials it provided (in most cases) a Spanish word that was phonologically similar to the stimulus item. Given that Chat GPT is trained on texts from a number of languages, including Spanish, it is perhaps not surprising that similar sounding Spanish words were presented as responses to the English nonwords (that were real words in Spanish). Such responses are technically correct; the response (in most cases) was indeed a word that sounded similar to the stimulus word. The response was just not an English word like the rest of the words that appeared in the prompt.

In the second part of this study, the same nonwords/Spanish words were presented again to Chat GPT using the more specific prompt “Give me an English word that sounds like <word>.” This resulted in most of the responses being English words that differed from the stimulus word by a single phoneme (e.g., added, deleted, or substituted), similar to the behavior of human participants in a comparable psycholinguistic task with either real words [15] or nonwords [14] as stimuli. (The responses provided in the first part of the study also tended to differ from the stimulus by a single phoneme/letter.) Although a computational analysis confirmed that each of the stimulus items had one or more English words that differed from it by the addition, deletion, or substitution of a single phoneme [16], a small percentage of the responses differed from the stimulus item by more than a single phoneme. A few of the responses were also repetitions of the stimulus item.

Previous work examining the ability of LLMs to generate a word that rhymes with the cue found that model performance lagged significantly behind the level of performance observed in humans [29]. In the present study, however, performance of the LLM in a different phonological task was comparable to human performance. Recall that words that rhyme must sound similar at the end of the word (e.g., light and right), whereas the present task left open the possibility that phonological similarity could occur anywhere in the word (e.g., light and like). Compared to the rhyme generation task, the additional freedom in responding that the present task allowed for may account for the better performance in the present study compared to previous studies looking at phonology in LLMs [29].

Although the task used in the present study may have been more flexible than the rhyme generation task used in previous studies [29], it is interesting that other ways to define phonologically similar (e.g., the word cat is embedded in the word catalog) were not observed in the Chat GPT responses. Different dimensions of phonological similarity have been observed in various psycholinguistic tasks with humans [15]. It is unclear what additional modifications to a prompt provided to Chat GPT would be required to elicit similar behaviors in the LLM.

Another behavior of Chat GPT in the present study is worthy of discussion, namely providing Spanish responses to English prompts as observed in the first part of this study. This language switching behavior is not unique to the present study, as indicated by reports from other users of Chat GPT in the OpenAI Developer Forum [34]. On the surface this language switching behavior in Chat GPT may resemble the phenomenon of code-switching in humans who know more than one language [35]. However, providing a response in a language that differs from the language used in the rest of the prompt is on some level inconsistent with (though not an egregious violation of) the unwritten and implicit expectations of human interlocutors to converse in a language(s) that is shared by both speakers. Although LLMs might be able to detect various patterns from the written input they are trained on, including the sounds that letters make in several languages, LLMs are not able to extract “rules” that are not written anywhere. These “rules” include the expectations and contextual knowledge shared by human interlocutors that allow them to communicate without language (e.g., a nod of the head), with non-linguistic noises (e.g., a grunt), using vague or context-dependent terms (e.g., “This one, not that one.”) or minimal linguistic input rather than elaborate, explicit, and engineered “prompts” during interactions.

Additional studies using the machine psycholinguistic approach and simple tasks such as the task employed in the present study may identify more domains, tasks, and abilities related to language and its flexible use that differ between LLMs and humans. Some developers may seek to reduce those gaps in performance to create LLMs that perform as well as or better than humans on language-related tasks. An alternative approach is to harness the differences between LLMs and humans to allow LLMs to complement the cognitive limitations of humans (as described in Study 1) or to allow humans to complement the inflexibility and lack of knowledge of unwritten rules and expectations in LLMs (as in the present study) to enable both entities to cooperatively produce a behavior that neither would be capable of by themselves.

Study 3: Nonwords varying in phonotactic probability

Study 2 and other work on LLMs [29] directly examined phonological similarity of (non)word stimuli to English words. In the present study, specially constructed nonwords were used to examine meta-phonological information about English words. That is, instead of generating an English word that was phonologically similar to the stimulus, Chat GPT was prompted to “subjectively” rate using a 7-point scale the extent to which specially constructed nonwords that varied in phonotactic information sounded like real English words.

Phonotactic information refers to the rules that govern which speech sounds are used in a language, and how those sounds can be arranged to form words in a language. For example, the “ng” sound is legal at the end of English words (as in the words ring and sing), but it cannot begin a word in English (however, in Albanian it is legal to start a word with that sound). Looking within a language at only the legal sounds and sequences of sounds, one sees that those sounds and sequences occur with varying frequency. This information is referred to as phonotactic probability [36], and it influences the ability of infants to distinguish between speech in their native language versus another language [37], assists children in acquiring new words [38], is used by infants [39] and adults [40] to segment words from fluent speech, and by adults to produce [41] and recognize spoken words [36].

Phonotactic information not only influences several language-related processes, but it has also been shown to influence other—seemingly unrelated—cognitive processes, such as deciding how much money one is willing to invest in a company that has a name that is easy or difficult to pronounce [42]. In another study human participants who were native speakers of English were asked to rate the wordlikeness and buyability of nonwords varying in phonotactic probability. Nonword stimuli that were rated as being more word-like were also rated as being more likely to be purchased if they were the name of a new product [43]. As noted in [43], brand names are similar to real words and proper names in that they all have a lexeme (i.e., the name) and a lemma (i.e., the sematic information or referent), but they differ from real words and proper names in that they also signal social status and may influence self-esteem. Thus, phonological information may affect more than just language-related processes [44].

The present study used the same nonword stimuli varying in phonotactic probability that were used in [43]. As in [43], Chat GPT was prompted to rate on a scale from 1 (corresponding to “Bad English word”) to 7 (corresponding to “Good English word”) the extent to which the nonword sounded like an English word, and then rate on a scale from 1 (corresponding to “least likely to buy”) to 7 (corresponding to “most likely to buy”) the likelihood of buying a product named <nonword > . The ratings provided by Chat GPT in both tasks were compared to the ratings provided by native speakers of English in [43].

Method

The nonwords used in the present study were the same 60 consonant-vowel-consonant, monosyllabic nonwords used previously in [43]. Phonotactic probabilities for the stimuli were calculated using the Phonotactic Probability Calculator [16]. In [43], nineteen native English speakers with no reported speech, language, or hearing disorders provided ratings to the nonwords. Those ratings were compared to the ratings provided by Chat GPT in the present study.

In the previous studies with humans, the stimuli were presented auditorily, and participants used a computer keyboard or response box to enter their numerical rating. Because Chat GPT does not directly process auditory input (i.e., spoken input is first converted to text), the phonological transcriptions of the nonwords were converted to orthographic representations using the English Sublexical Toolkit [28] to ensure consistent mapping between spellings and sounds. Spellings were adjusted until each item had 100% spelling-sound mappings as per the English Sublexical Toolkit [28].

From 7–12 August 2024 the nonwords were pseudo-randomly presented to Chat GPT 4o, first with the prompt: On a scale from 1 (corresponding to “Bad English word”) to 7 (corresponding to “Good English word”) how well does <nonword> sound like an English word? After all 60 items had been rated for wordlikeness, the nonwords were presented in a different pseudo-random order with the prompt: On a scale from 1 (corresponding to “least likely to buy”) to 7 (corresponding to “most likely to buy”) how likely would you buy a product named <nonword > . The same wordlikeness and buyabilty prompts were used in [43].

Results

As reported in [43], subjective human ratings of wordlikeness correlated significantly with both objective measures of phonotactic probability: sum of the segments (r = .55, p < .0001), and sum of the biphones (r = .56, p < .0001). The subjective human ratings of wordlikeness correlated with the subjective human buyability ratings (r = .70, p < .0001). The subjective human ratings of buyability correlated significantly with both objective measures of phonotactic probability: sum of the segments (r = .51, p < .0001), and sum of the biphones (r = .48, p < .0001).

In the present study, the same analyses (using [24]) were performed with the responses produced by Chat GPT. Ratings of wordlikeness from Chat GPT correlated significantly with both objective measures of phonotactic probability: sum of the segments (r = .35, p < .0001), and sum of the biphones (r = .35, p < .0001). The ratings of wordlikeness from Chat GPT correlated with the buyability ratings from Chat GPT (r = .68, p < .0001). The ratings of buyability from Chat GPT correlated significantly with both objective measures of phonotactic probability: sum of the segments (r = .51, p < .0001), and sum of the biphones (r = .50, p < .0001).

Because the stimuli presented to Chat CPT were converted to orthographic representations, the same analyses were repeated conditioned on the number of letters in each nonword, reading consistency (p(P|G) from [28]), and spelling consistency (p(G|P) from [28]). Again, ratings of wordlikeness from Chat GPT correlated significantly with both objective measures of phonotactic probability: sum of the segments (r = .43, p < .0001), and sum of the biphones (r = .42, p < .0001). The ratings of wordlikeness from Chat GPT correlated with the buyability ratings from Chat GPT (r = .68, p < .0001). The ratings of buyability from Chat GPT correlated significantly with both objective measures of phonotactic probability: sum of the segments (r = .48, p < .0001), and sum of the biphones (r = .45, p < .0001).

Paired samples t-tests were used to compare the wordlikeness ratings obtained from humans to the wordlikeness ratings obtained from Chat GPT, and the buyability ratings obtained from humans to the buyability ratings obtained from Chat GPT. For the wordlikeness ratings, Chat GPT produced significantly higher ratings (mean = 3.94; sd = 1.32) than humans (mean = 3.53; sd = 0.76; t (59) = −2.78, p < .01). Similarly, for the buyability ratings, Chat GPT produced significantly higher ratings (mean = 3.88; sd = 1.06) than humans (mean = 3.28; sd = 0.63; t (59) = −5.06, p < .0001).

Discussion

In the present study, a cognitive/psycholinguistic task used with humans in [43] was used with Chat GPT. That is, Chat GPT was prompted with specially created nonwords that varied in phonotactic probability, and instructed to use a 7-point scale to rate how much the stimulus resembled English words and how likely one would buy a product with the name of the nonword. Despite the nonwords being transformed to an orthographic form (i.e., a string of letters) in order to present them to Chat GPT, the performance of the LLM resembled the performance of humans in both rating tasks. That is, the “subjective” ratings of wordlikeness correlated with two measures of phonotactic probability (i.e., the sum of the segments, and the sum of the sequences of segments), the ratings of buyability correlated with two measures of phonotactic probability, and the ratings of wordlikeness correlated with the ratings of buyability. These relationships remained even when the correlations were conditioned on the number of letters in each nonword, and two measures of the consistency in mapping letters and sounds.

The only difference between the ratings provided by humans and by Chat GPT was that the wordlikeness ratings from Chat GPT tended to be higher than those provided by humans. This difference may be due to humans using (on average) a more restricted range of the rating scale (minimum mean rating = 2.32; maximum mean rating = 5.47) compared to the range used by Chat GPT (minimum rating = 2; maximum rating = 7; N.B., this may also be an artifact of sampling and statistical measurement), or to humans using factors not used by Chat GPT in assigning their ratings (e.g., “eccentric phonic essence”), or to any number of unmeasured characteristics in the present study.

The ability of an LLM to perform like human language users in assessing meta-phonological information (i.e., how word-like a stimulus sounds) despite receiving only text as input during training may appear at first to be an impressive accomplishment. However, even languages like English with deep orthographies [17] have consistencies and regularities in the mappings between phonemes and letters that can be described probabilistically and potentially exploited [28].

There are many other statistical regularities that have been observed in human languages. One example is Menzerath’s law, which states that an increase in a particular linguistic unit is associated with a reduction in the size of its constituents. Thus, a long sentence (as measured by the number of clauses) will tend to have shorter clauses. Similarly, a long word (as measured by the number of syllables) will tend to be comprised of small or simple syllables (such as CVs or Vs). The relationship between unit and constituent size has also been observed in music and genomes [45]. There are numerous statistical regularities in human languages observed by [33], such as the correlation between word-length and word frequency (i.e., long words tend to occur less often in the language). Also, words that occur often in the language tend to be phonologically or orthographically similar to many other words in the language (e.g., [46]). Given that rodents can be trained to exploit certain regularities in human speech to distinguish between phoneme contrasts in English [47], the ability of a machine that is trained to discover and exploit the numerous regularities found in human languages now appears less impressive.

The results of the present study using the machine psycholinguistic approach show that Chat GPT can discover and exploit regularities in English words to perform a nonword rating task at a level comparable to humans. Using AI to discover these and other regularities in human languages could prove useful to researchers developing natural language processing (NLP) applications and augmented and alternative communication (AAC) devices that enable humans to accomplish more than they would be able to without those applications or devices.

Note, however that the regularities found in English words described above relate to the phonological or orthographic characteristics of the words. There are, of course, regularities in human languages that relate to the semantic characteristics of words, such as the observation that words that occur more often in the language tend to have many meanings [33; 48]. Therefore, in Study 4 we consider the performance of Chat GPT on a semantic task.

Study 4: Create a new English word for a novel concept

Although some previous work found similarities in how humans and LLMs respond, other previous studies using the machine psychology/psycholinguistic approach have found important differences in the responses generated by humans and the responses generated by LLM. In a semantic associate task (i.e., What’s the first word that comes to mind when you read the word _____?) human generated associates contained less gender bias and were richer and more varied than the associates generated by an LLM [49]. For example, in response to the prompt dog, humans gave cat and other syntagmatically related associates (e.g., love, bone, furry, leash, loyal), whereas the LLM tended to provide associates that were paradigmatically related (e.g., puppy, canine, animal, pet). Understanding such differences in performance may prove useful in determining how best to use AI to complement human performance in a variety of domains.

In the present study Chat GPT was prompted to create a new English word to refer to a novel concept described in the prompt. This task differs from the task used in Study 1 where a definition was requested for a novel word that was presented in the prompt. The task used in the present study enabled us to examine how various types of linguistic information—phonological (perhaps morphological) and semantic information—might interact and be used in a novel and creative way (for examples of the creative use of language in humans see [50,51]). Google Books Ngram Viewer [20] and dictionary searches were used to confirm the novelty of the new word responses.

Method

Table 2 shows the prompts used in the present study. The prompts were presented to Chat GPT 4o in the order listed in the table on 19-AUG-2024. The new word that was created is also presented in Table 2.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. The prompts presented to Chat GPT, and the novel word produced as a response.

https://doi.org/10.1371/journal.pone.0325612.t002

Although there is a German word (schadenfreude) for the concept described in prompt 1, there currently is no English word in contemporary usage for that concept. The German word schadenfreude is sometimes used by English speakers, however. Similarly, there is a Yiddish word (naches; typically applied to one’s family) that is close in meaning to the concept described in prompt 2, but there currently is no English word in contemporary usage for that concept.

General discussion

Nonsense, in the form of specially created sequences of letters or sounds that lack semantic content, has played an important role in the cognitive and language sciences as a tool to probe the implicit knowledge that human language users represent in memory as well as the processes used to retrieve those representations. In the present set of studies, nonwords of various kinds were presented to ChatGPT with prompts that resembled the instructions of tasks previously used in psycholinguistic experiments with humans. In some of the studies the responses from ChatGPT were compared to previously collected data from human language users to better understand the performance of ChatGPT in those tasks. Future studies using the machine psycholinguistic approach to understand how the behavior of ChatGPT resembles and differs from the performance of human language users as well as the type of errors each entity makes may lead to advances in the development of LLMs and other forms of AI that enhance rather than replicate human cognitive and linguistic abilities.