Dataset

The corpus was collected using X (formerly Twitter, using application programming interface standard search, publicly accessible via API V2 on the academic research access level) and Reddit (using Pushshift API, adhering to fair use non-commercial academic research). All English-language content containing “prompt engine*” as a keyword, including inflectional variants on “engine”, like engines, engineer(s), engineering, engineered, used in context, not case-sensitive, and either as a keyword or hashtag, posted within the span of a year (1 May 2023–30 April 2024) were collected (n = 298,774). A summary of the methodology is provided in Fig 1.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 1. Methodological Summary: Data Collection and Analytic Plan.

https://doi.org/10.1371/journal.pone.0351077.g001

Research Direction 1: Prominent Peak Modelling

Of our corpus dataset, daily mention counts (defined by the total number of original tweets, posts and replies containing the “prompt engine*” keyword) were charted. Using this data, prominent peak dates in which the keyword experienced a significant spike in its count were identified using peak prominence detection modelling package scipy.signal.find_peaks (Python version 3.11: SciPy 1.11.1). This methodology is attested in prior published papers charting burgeoning public perceptions of ChatGPT [20], using prominent peaks as a proxy signal for statistically-significant spikes in interest and engagement in the topic. The study design leverages emerging coinages to study novel phenomena and identify key drivers of buzz in a process of socially-derived agenda setting [21] and reflective of opinions circulating a societal microcosm [22].

The package module calculates for every date’s mention-count value (A), the lowest possible point prior to that date (B) and after that date (C): the average of these values (A-B; A-C) defined as the date’s prominence score, a value assigned relative to the lowest possible point on either side of the calendar. Of 97 candidates for statistically-significant peaks throughout the study period (x = 615.1, σ = 1507.5), prominence scores of said peaks ranged from 71 (lower quartile), 265 (median), 568 (upper quartile). A data-driven approach was used to determine an adequate threshold for significance: peaks that appeared within the 95^th percentile of prominence (i.e., prominence> 1,500) were selected for further investigation, yielding 6 significant peak dates. All posts published on these dates were then qualitatively interrogated by sampling the top-100 most-engaged posts (defined by an aggregated cumulative score of likes, replies, comments and reposts in the form of quotes) to identify the main content themes and discussions that led to an increment in online discussion. Thematic categorization was first conducted by two independent investigators, thereafter, reconvening to evaluate qualitatively-coded findings. Initial inter-coder agreeability was at 89.83%, while consensus-resolved agreement reached 96.67% of all sampled posts, with a remaining 3.33% of posts persisting in inter-coder nuanced disagreement, particularly for posts that tended to be about two topics at once. Investigators considered various conversational threads and opinions, taking care to note detractors (often in the form of rebuttal replies, or comments receiving high upvote volumes) and supporters (often in the form of agreeable replies, and large amounts of likes or upvotes) where applicable.

Research Direction 2: Quarterly Conversational Metrics

The following content metrics used in conjunction with “prompt engineering” posts were amalgamated on a quarterly basis. First, each post was categorized as positive, neutral, or negative, with valence scoring using a fine-tuned transformer-based sentiment model roBERTa adapted from Google’s open-source pre-trained Bidirectional Encoder Representations from Transformer (BERT) for deep representations from unlabeled text through joint conditioning left and right context [23], most widely employed and cross-validated academically in the analysis of online sentiment analysis [24]. Alongside that, we present the highest-frequency bi-grams and top emojis per quarter. Top bi-grams were identified by raw count after text was tokenized into contiguous two-word sequences with the manual exclusion of stop-words, defined as auxiliaries containing little-to-no semantic information, for example, determiners, prepositions and conjunctions were omitted from qualifying for this list. Top emojis were identified by raw count attested in the dataset with reference to standardized Unicode emoji libraries and presented in order of highest count. This charts the shifting chronology in opinion, content foci, and prevailing social buzz surrounding this topic.

Prominent Peaks and Emergent Themes

A total of 6 prominent peak dates (i)–(vi) were identified and summarized in Fig 2, accompanied with a thematic discussion of each spike’s discourses.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 2. Prominent Peaks in “Prompt Engineering” Conversations across 300,000 Social Media Posts from 1 May 2023 to 30 April 2024.

https://doi.org/10.1371/journal.pone.0351077.g002

Peak (i): Validity of the Skill

This first peak date saw an uptick in discussions on the legitimacy of the title “prompt engineer”. Ongoing conversation saw differing opinions on the topic, consisting of three main opinion camps. First, detractors argued that prompt engineering was neither a real skill nor legitimate profession, and an over-complicated way of describing a simple process of typing text parameters into a large language model’s interface. Others mocked “prompt guides” produced by self-proclaimed “prompt gurus”, insinuating that the skill was “overhyped” to market productivity courses for personal profit. This camp pointed out that these platforms are themselves already a product designed for end users and did not require further hand-holding to use, as all it required was asking questions in English.

On the other hand, supporters believed that prompt engineering required legitimate training (e.g., precise word choices, streamlined questions, providing relevant code snippets, designing tasks as efficiently and clearly as possible, using fewer tokens to reduce confusion, identifying ideal prompt patterns, using “jailbreaks” or “prompt injections” with ingenuity to bypass filters) to elicit the best results (accurate responses, desired information, free from false information or “hallucinations”). Supporters highlighted problem formulation as an enduring and adaptable skillset, especially in knowledge-based jobs (e.g., finding information, assessing its viability, highlighting critical findings): fundamental components of the research workflow. Many pointed out that tech-illiteracy was still rampant in many industries, providing anecdotes of colleagues who were incapable of using search engines to find desired information, lacking the technological abilities to access basic tools, or unable to frame their search requests cogently: these disparities necessitating the skillset of the prompt engineer.

A third camp debated the semantics of “engineer”, perceiving that it was “pretentious” and an injustice to actual certified engineers, who had to complete a 4-year degree and receive proper accreditation to earn the title. This camp opined that more accurate titles should have been adopted, ranging from neutral ones like “prompter” or “prompt tweaker, crafter, specialist, developer or technician”; to denigrative, like “prompt monkey” or “code jockey”.

Peak (ii) Developments and Future Ramifications

This discussion spike was attributable to several field updates. First, early access to “PromptIDE”, an “integrated development environment for prompt engineering and interpretability research” was announced by artificial intelligence company xAI. Second, artificial intelligence research organization OpenAI held their first developer conference “DevDay”, announcing custom GPTs with pre-customized prompts, made available for paid service users. These announcements generated discussions among AI enthusiasts, ranging from excitement over potential new tools, computational infrastructure and processing power; though other commenters tempered their expectations given that only a handful of screenshots and descriptions were included in the initial announcements. Other commenters suggested that these developments may affect small startups and independent bootstrappers dabbling in creating AI wrapper applications, given overlapping scopes. Others posited that with enough time and iterations, models would prove more capable than a human prompter in deciphering and crafting custom prompts, rendering the engineer’s role obsolete.

Peak (iii): Longevity of the Title

This spike date occurred in conjunction with the circulation of OpenAI’s prompt engineering guide, containing strategies for obtaining optimal results. This spurred discussions on the usefulness of large language models, especially in coding and menial tasks, with the general consensus conceding that the technology had not lived up to its hype and initial promise of task automation and general all-purpose toolkit functionality. The viability of a prompt engineering “career” was also called into question: many pointed out the lack of job security and current limitations to the technology, suggesting that artificial demand created for the position would be phased out.

Peak (iv): Allegations of Artist Attribution Infringement

This notable spike was attributed to a revelation that generative AI image-prompt program Midjourney developers were caught training their AI models on a database of 16,000 artists, whose artworks were scraped without consent. This information, submitted as evidence for an ongoing lawsuit, sparked outrage and condemnation at the blatant art theft. There was a clear discursive distinction made between individual human artists, who respectfully and organically learn from other inspiring artists, by replicating techniques to hone their skills to creatively express themselves, and self-proclaimed “AI prompt artists”, whose outputs were perceived as “lazy” and simply piggybacking off stolen amalgamated art. Many championed the use of novel digital tool “Glaze” [25] to sabotage future art scraping, suggesting that artists apply “style cloaks” before uploading their art online.

Peak (v): Biases Encoded in Training Data

This peak date involved conversations on the transparency of in-house prompt engineers in shaping AI behavior, especially in fine-tuning models to reduce potential biases. Given that part of the data used to train models are drawn from the internet, an inherently “poisoned well”, containing unrestricted or unverified information, inevitably containing biases, hate speech, “polluted” with prejudices and stereotypes, discussions surrounded the need for high-quality training data, better curation and sanitization of large datasets to allow nuanced, respectful outputs authentically representing human society. However, this idealized workflow comes at the cost of being expensive, labor-intensive, and time-consuming.

Instead, commenters speculated that engineers were using pre-prompt instructions to correct for potential biases (e.g., adding keywords or removing sensitive words from the original prompt keyed into the user interface on the backend, blanket refusal of requests that may produce biased results). This method was perceived as “lazy”, “slapdash”, “ineffectual”, “wallpapers” over real problems, and only adds “token diversity” after the fact, given that the model still contains data pollution that can be easily circumvented using jailbreaking and prompt injection methods. Unintended consequences of being overly restrictive may surface (e.g., a parameter intending to encourage diversity may strongly emphasize minority representation and over-compensate by removing all other individuals from an output), suggesting a need for more guardrail transparency (i.e., documentation of guardrail topics, design ethos, specific filtering commands and code, reinforcement learning iterations and qualitative discussions on modifications made).

Peak (vi): An analogy for Raising a Child

This biggest peak stemmed from an original tweet from OpenAI co-founder Elon Musk, who opined that raising children was akin to “18 years of prompt engineering”, which sparked thousands in engagement metrics. Some supporters of this analogy agreed that a brain was akin to a machine learning algorithm, given that artificial neural networks were conceptually modelled after neurons. On the other hand, more were of the opinion that child-rearing was nothing like prompt engineering, as it is a complicated social process involving years of teaching, guiding and inspiring a sapient being to learn and grow, intertwined with lessons on morality and appreciation for the world. Many took the opportunity to critique a larger societal narrative tied to Silicon Valley, pointing out the inhumane oversimplification of comparing raising a child to a ChatGPT session and the irrationality of “tech bros” who speak about childcare like running diagnostics on a robot.

Content Metrics

Salient content metrics were identified on a quarter-on-quarter basis (wherein each quarter’s keyword count is as follows: Q1 n = 73,326, Q2 n = 53,444, Q3 n = 93,074, Q4 n = 78,930) and summarized in Fig 3 and elaborated further in 5 key findings. Finding 1 highlights shifts in valence scoring, finding 2 highlights predominant themes encoded in highest-frequency bi-grams and emojis embedded in tutorial posts, and findings 3–5 elaborate on high-frequency bi-grams containing notable conversational buzz topics surrounding prompt engineering discussions.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 3. Content Metrics of ‘Prompt Engineering’ Conversations across 300,000 Social Media Posts from 1 May 2023 to 30 April 2024.

https://doi.org/10.1371/journal.pone.0351077.g003

Finding 1: Increasingly Negative Valence

Over the study period, the negative valence of “prompt engineering” posts increased from 14.1% in Q1 to 37.3% in Q4. Conversely, positivity percentages decreased from 36.4% in Q1 to 22.8% in Q4. Overall, this suggests waning positive sentiments toward the field over the course of a year.

Finding 2: Predominant Tutorial Posts

Throughout the dataset, there was a marked prevalence of posts marketing prompt engineering tutorials, courses, guides, and individuals championing the field as “the next big thing”. Of the highest-frequency bi-grams, a majority contained an educational slant. For instance, bi-grams connoted to informative material (“new courses”, “new audiobook”, “online courses”, “free guides”), best techniques (“biggest skill”, “untapped method”, “problem formulation”, “engineering techniques”) and the potential impact on career and work (“job openings”, “productivity tips”). Overall, such posts advertised resources purported to transform one’s efficiency, reflecting a broad surge in the circulation of prompt-engineering guides online.

Similarly, highest-frequency emojis conveyed informative sources (🧵, 🔗, 📂, 📝, 📚, 🎓, 🧠, 🔖, 📢: the symbols of the thread, hyperlink, file folder, memo, books, academic cap, brain, bookmark and megaphone respectively), technological potential (🚀, 🤖, 🌐, 🌍, 💻, 📊, 📈: symbols of the rocket, robot, world wide web, globe, laptop and statistical charts respectively), call-to-action pointers toward cardinal directions (👇, 👉, ➡), toolkit (🔨, 🛠, 👷, 🔎: symbols of the hammer, construction tools, construction worker and magnifying glass respectively), various expressions of excitement and positivity (🆕, ✨, ⭐, 🌟, 🔥, 🎉, 🔮, 💡, 👍, 👏, 🙌, 🤯: symbols of newness, stars and sparkles, party popper, crystal ball, lightbulb, thumbs up, applause, raised hands celebrating success, mind-blown respectively). Several instances of doubt were displayed in emojis like 🤔 (thinking face), 🤷 (shrug), 😂🤣 (laughing-crying symbols), 😱 (fear).

Finding 3: Multiple Instances of Humor

Present in Q1 was the bi-gram “greatest prompt”. The context of the original post had a mocking tone, with the poster claiming to be the “greatest prompt engineer” as they had prompted ChatGPT to rank U.S. presidents on their “absorbency” and managed to circumvent the chatbot’s initial resistance to answering the question given its absurdity and political nature. Through the use of hypothetical scenario prompting, the output ranked the absorbent quality of George Washington’s “powdered wig” against Theodore Roosevelt’s “robust moustache”. The post garnered 9.3 thousand likes and was an example of users deliberately eliciting humorous replies using prompt engineering while engaging with large language models.

Also present in Q2 was the bi-gram “documentary vibes”. The context of the original post mocked generative AI users for claiming to be “artists” when all they did was type words into an interface using such prompts as “award winning photograph”, “beautiful Rembrandt lighting”, “extreme fine art photography (SUBJECT) in the style of (PHOTOGRAPHER)”. The crux of the commentary hinged on whether such self-proclaimed “art prompt engineers” were displaying artistry, or simply “stealing” from the work of established artists. The post garnered 20 thousand likes and was an example of users discussing the ethics behind artistic integrity.

Present in Q4 was the bi-gram “no yapping” (vernacular for being excessively verbose). The context of the original post jokingly suggests that this phrase was the perfect “pro-level” prompt engineering strategy, contrary to an existing litany of overly-complex rules that prompt engineer “gurus” tout online. The poster mocks existing guidelines and subverts the expectation of precise prompt construction with the use of colloquial slang. The post received 15 thousand likes and was an example of users poking fun at the current prompt engineering landscape, pointing out the latent absurdity of taking the field too seriously.

Finding 4: Legal Implications of Using AI Chatbots

Present in Q3 was the bi-gram “procurement specialist”. The context of the original post was an individual conversing with a car dealership’s AI Assistant, demonstrating how simple it was to use specific prompts to get the chatbot to make promises the company could not honor. Specifically, the original post claimed to feed prompts into the chat window to obtain a car for $1.00 USD, managing to receive an affirmative response that read “that’s a deal, and that’s a legally binding offer”. The chat bot was later deactivated. The post garnered 101 thousand likes, emblematizing the ease at which bots could be manipulated using specific prompts to promise consumers items they had no apparent authority over.

Similarly, present in Q4 was the bi-gram “chatbots hallucinate”. The context of the original post cited a recent international airline having to honor a policy invented by its chatbot after it provided inaccurate information on bereavement refunds to a passenger. The poster jokingly suggests for other passengers to use prompt engineering to make airline chatbots hallucinate in a similar way to offer “$1 first class tickets”. The inciting post sparked discussions on the ease at which consumers could elicit perks using prompt loopholes, and whether corporations should be responsible for ensuring information across their AI chatbots or webpage assistants was accurate to avoid potential financial liabilities.

Finding 5: Stylistic Impacts

Present in Q2 was the bi-gram “unnecessary anthropomorphism”. The context of the original post claimed that the process of prompt engineering was akin to a keyword search in a library, and that prompters ought not to erroneously imagine they are engaging with a human when using large language models. This post invited discussions on the validity of this claim, given that such models were trained on natural language written by humans and for humans, thus making it valuable to converse with it like a human. For instance, adding “please” and “thanks” to prompts, or using compassionate directives like “take a deep breath and proceed to work on this step-by-step” were alleged to produce better results, lending credence to simulating anthropomorphism when framing tasks linguistically. The long-term implications on linguistic mannerisms in online writing were also surfaced, with some reporting that large language model ChatGPT had an identifiable uncanny “accent”: often too polite (e.g., using too many phatic tokens, being tonally-obsequious or sycophantic, prone to empty flattery), framing answers vaguely and avoiding voicing critique toward the prompter.