Introduction

The CrossFit^® Open (CFO) has acted as the initial online qualification round for the CrossFit^® Games competition since 2011 [1]. The 3–5-week CFO has typically consisted of 4–6 workouts that variably challenge an athlete’s weightlifting strength, gymnastic skill, and endurance capacity [2, 3]. The details of each workout are announced weekly, and competitors may rely on the fact that all workouts within a particular year will be unique to each other. However, except in 2011 and 2022, each CFO competition has included a workout drawn from a previous competition, with nine distinct workouts acting as repeats between 2012 and 2021. Adapted from [2], the composition and movement standards for these workouts are presented in Tables 1 and 2, respectively.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Repeated CrossFit^® open workouts.

https://doi.org/10.1371/journal.pone.0283910.t001

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Exercise movement standards.

https://doi.org/10.1371/journal.pone.0283910.t002

After their introduction, CFO workouts become benchmarks (i.e., named workouts that are commonly known across training facilities) that may be integrated into training. Although the CrossFit^® strategy aims to constantly vary the training stimulus to promote simultaneous and generalized fitness improvements [4], benchmark workouts will appear more often throughout training. Thus, it is reasonable to expect that greater familiarity with these workouts would lead to improved scores whenever one is officially repeated in competition [5, 6]. As one becomes more familiar with the physical tasks appearing in sport, they learn to activate more muscle when given the same tasks and eliminate inefficient and unnecessary actions [7–9]. Greater experience affords more opportunities to develop relevant skills and strategies that affect one’s approach to a workout (i.e., pacing strategy) and its resultant physiological demand. This is relevant to CrossFit^® competition because workouts are commonly structured to produce a score that emphasizes maximal workout density. Athletes are scored by their time-to-complete (TTC) assigned work or they are tasked with completing ‘as many repetitions as possible’(AMRAP) of assigned exercises within a set time limit [4, 10]. Efficiently performing movements and transitioning between exercises is at least partially responsible for an individual’s ability to maximize their score, and this ability can only be developed with practice. However, aside from documented record performances from the sport’s top athletes in repeated CFO workouts [2, 3], it is unclear whether improvements have occurred throughout the remaining competition pool.

Another important consideration is the impact of increased participation, particularly within the women’s division, on performance and ranking. The initial CFO pool of 26,000 athletes consisted of 13,127 Rx competitors (i.e., non-scaled or age-group athletes who completed all CFO workouts as prescribed) and 4,506 people competed in the women’s division (34.3%), and that number grew to 42,799 in 2021 (36.7%) [11]. Though the same absolute rank becomes mathematically less important with more people vying for the same number of spots at the Games, the effect on percentage-rank remains unclear. It is also unclear whether such an effect, if present, has been equal across the men’s and women’s competitive divisions.

Unlike more traditional sports, the only observable difference between the men’s and women’s divisions are participation and programming [12]. With programming, workouts are often scaled, presumably to account for known physiological differences between biological men and women [2, 3]. However, scaling practices are not uniformly prescribed and their effect on performance has not been specifically explored. Among repeated CFO workouts, weightlifting intensity loads were the most often scaled programming feature with loads assigned to women ranging between 62.2% and 73.3% of those assigned to men, including wall ball shot target distance and box jump height [2]. While scaling resistance-based exercises may account for known strength and power differences [13], this rationale has not been extended to gymnastic-calisthenic exercise prescription (e.g., handstand push-ups/walking, burpees, pull-ups, and muscle-ups), though difficulty may be naturally equated for these. Gymnastic-calisthenic exercises primarily require the athlete to maneuver their own body mass about an object (e.g., pull-up bar, rings, box) [4, 10] and women tend to possess less body mass [13, 14]. In contrast, scaling has yet to be applied to jumping rope and rowing tasks, which seems to ignore known sex differences in physiological measures of cardiorespiratory fitness [13, 15]. It remains unknown whether these programming decisions have been successful in equating the challenge presented to competitors in the men’s and women’s divisions, nor if this has changed over time.

In non-CFO workouts, one study reported no differences between 13 men and 10 women for a 20-minute AMRAP, as well as a TTC version of the same workout, that scaled rowing calories, deadlift loads, and kettlebell swing loads but not burpees [16]. However, the small sample size affected the generalizability of those findings and experience (the only available proxy of athlete skill) was significantly different between sexes and could have affected how each approached the workouts. Conversely, normative scores were developed from very large random samples of CFO competitors (n = 7,046–89,792) for all workouts programmed between 2011 and 2022 [17], and performance differences between men and women were noted in 56 out of 60 total workouts. Although this implies ineffectual CFO scaling, it would be premature to definitively make this conclusion due to unequaled sample sizes and participant skill. Equating these would provide a better understanding about whether scaling practices have led to similar scores in the men’s and women’s divisions. In turn, this information would assist in developing generalized training recommendations and provide better context for studies aiming to determine physiological predictors of performance. Thus, the purpose of this investigation was to begin that process by examining performance differences between similarly-ranked competitors within the men’s and women’s divisions, and their changes over time, in repeated CFO. A secondary aim was to determine how changes in performance translated to ranking in each workout in each workout and the overall competition.

Methods

Participants

Based on pilot data [18] and the expectation of a small effect (Effect of f = 0.10), an a priori analysis using G*Power (v. 3.1.9.7, Heinrich-Heine-Universität, Germany) for a repeated-measures design indicated at least 328 participants would provide sufficient power (α = 0.05, β = 0.95) to observe differences between divisions and performances in each repeated CFO workout. Since collecting data on the same 328 men and women from 2011 to 2021 would produce a very specific sample, one that is likely representative of a subset of CFO participants, this study identified cases based on percent rank and made comparisons between athletes of the same percent rank across competition years.

Initially, a stratified list of 500 percentile ranks were identified based on the approximate percentage of cases that would fall within each standard deviation (SD) bin: 0.0–0.5 SD (38.2%, n = 192), 0.5–1.0 SD (30.0%, n = 150), 1.0–1.5 SD (18.4%, n = 92), 1.5–2.0 SD (8.8%, n = 44), 2.0–2.5 SD (3.4%, n = 16), 2.5–3.0 SD (1.0%, n = 4), and 3.0–3.5 SD (0.2%, n = 2) [19]. Ranks were evenly divided within each bin and across each side of the mean (e.g., –2.75 SD, –2.5 SD, +2.5 SD, and +2.75 SD). These percent-ranks would be used to identify the specific cases within each CFO year that would be drawn from the pool of athletes who also met study criteria. Age, rank, and performance data were retained for all athletes, between the ages of 18 and 54 years (i.e., non-age group athletes), who completed all CFO workouts as prescribed (i.e., as Rx with no within-division scaling) within a specific competition year. Additionally, cases were excluded if they did not complete at least one round (in AMRAP-style workouts), the first exercise couplet (in repeated couplet workouts), or all repetitions assigned for the first exercise (for TTC or when multiple rounds were not expected) for every workout within a single competition year. These criteria were meant to limit the inclusion of workout “specialists” and those who did not intend on completing or could not perform the exercises for the Rx workout. To match specific cases with the identified percent ranks to be drawn from within each year’s athlete pool, retained athletes were ordered based on their final within-division overall competition rank and then assigned a within-year percent rank (i.e., percent rank among athletes meeting study criteria). The final within-year percent ranking was used to identify 500 athletes within each year who would be included in this study. Thus, the same array of percent-ranks was represented by the athletes retained each year. Table 3 provides a summary of the initial pool of competitors for each year, the number of cases meeting study criteria, and the age and final competition ranking characteristics of those retained for analysis.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Case selection and sample characteristics.

https://doi.org/10.1371/journal.pone.0283910.t003

Results

Ranking and athlete age

No differences were seen in the workout percent-rank calculated for each repeated workout within each comparison, except for a main effect for sex division when comparing 12.4 and 13.3 (F = 17, p < 0.001, η²_p = 0.02). Workout percent-rank was greater in men (29.8 ± 27.1%) than women (27.1 ± 23.5%) during the workout’s original and repeated appearances. In contrast, significant time x sex division interactions (F = 14–1089, p < 0.001, η²_p = 0.01–0.52) were noted with original overall rank for each workout comparison. In each comparison, men ranked lower (i.e., farther from first place) within their division than women and ranking further declined whenever a CFO workout was repeated. Likewise, the original rank assigned for performance in each workout followed a similar pattern in each comparison.

Significant time x sex division interactions were only found in the comparisons between 11.1 and 14.1 (F = 4, p < 0.05, η²_p < 0.01) and between 11.6, 12.5, and 18.5 (F = 3, p < 0.05, η²_p < 0.01). Average age in men increased from 2011 to 2012 (+1.1 years, p < 0.05), from 2011 to 2014 (+ 2.0 years, p < 0.001), and from 2012 to 2018 (+1.8 years, p < 0.001), but only from 2011 to 2018 in women (+1.5 years, p < 0.001). Main effects for time, where age increased in subsequent years, were noted in all remaining comparisons except for between 14.5 and 16.5 (p = 0.053). Main effects for sex division were also noted with the comparisons between 14.2 and 15.2 (F = 9, p < 0.01, η²_p = < 0.01), 14.5 and 16.5 (F = 5, p < 0.05, η²_p < 0.01), and 16.2 and 19.2 (F = 6, p < 0.05, η²_p < 0.01), where men were older than women in each case. Comparisons between sex divisions across repeated CFO workouts for age, overall rank, workout rank, and workout percentile rank are presented in Table 4.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 4. Comparisons between age, overall rank, workout rank, and workout percent rank in repeated CFO workouts.

https://doi.org/10.1371/journal.pone.0283910.t004

Workout completion rate

Significant time x sex division interactions (F = 14–357, p < 0.001, η²_p = 0.01–0.26) were observed for all workout completion rate comparisons except for between 18.4 and 20.3, where no differences were found between sex divisions or repeated performance. Comparisons between sex divisions and across repeated CFO workouts for repetition completion rate are illustrated in Fig 1.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Repeated workout comparisons for repetition completion rate.

Note: * = Significant (p < 0.05) difference between men and women; # = Significant (p < 0.05) difference from previous year(s).

https://doi.org/10.1371/journal.pone.0283910.g001

Discussion

This study examined performance differences between competitors within the men’s and women’s divisions in repeated CFO workouts and their effect on competition ranking. Out of nine separate repeated CFO workouts, performance improved in five workouts, diminished in two, and remained the same in one (i.e., the 9-minute AMRAP programmed for 18.4 and 20.3). Though their workout pace generally improved over time, the absolute rank awarded to competitors on the Leaderboard [11] declined. An outcome most likely due to increased participation over time because neither the competitors’ percent-rank overall nor within repeated workouts changed. Thus, to achieve a comparable or better percent-rank in a repeated workout, athletes would have had to improve performance beyond what was typically seen across all competitors. Additionally, initial, and subsequent performances were not equal among men and women. Men were initially faster in four of the workouts, while women outpaced them in two, and tied in the remaining three. Men were still faster in more workouts when they were repeated (3 workouts versus 2 workouts), but the specific workouts were slightly different. Men and women tied on the second iteration of two workouts where men were initially faster, while men were slower than women in a third workout where they had been initially faster. Women also tied men on the second iteration of a workout where they were initially faster. Aside from a preliminary conference presentation [18], this is the first study to investigate performance changes in official CrossFit^® competition workouts.

Improved performance was observed in six of the nine repeated CFO workouts. A learning effect would seem to be the most likely explanation for these improvements. Except when they are repeated, CFO workouts are novel and uniquely vary in exercise complexity, relative intensity, workload requirements, and structural design (i.e., AMRAP, TTC) [2, 3]. Though experience may facilitate the process [22–25], competitors may not be able to reconcile their own physiological and skill-related abilities with specific workout requirements on their first attempt. Moreover, any extra attempts would contribute to accumulated fatigue and damage that is not likely to have abated within the typical 4-day allotment [3, 20, 26], and this could negatively impact later efforts. Athletes may simply not find their ideal pacing strategy for each workout within a single CFO competition. However, after their introduction, CFO workouts may be integrated into training and/or frequently discussed (in-person or online) amongst athletes. With an average of 2.4 years separating repeated CFO workouts [5, 6], athletes have ample opportunities to refine their pacing strategy. Albeit they would not know which and when specific CFO workouts might be repeated, leaving an ever-growing list of potential workouts and skills in need of practice.

Having time to perfect pacing strategy may have been less relevant to performance in the three workouts that did not improve (i.e., 16.2–19.2, 16.4–17.4, and 18.4–20.3). Compared to all other workouts, these three required more strength and capacity to lift heavier loads or perform complex gymnastic movements for multiple repetitions. For instance, deadlift loads ranged from 225–315 lbs. (102.1–142.9 kg) for men and 155–205 lbs. (70.3–93.0 kg) for women, and power clean loads ranged from 135–315 lbs. (61.2–142.9 kg) for men and 85–205 lbs. (38.6–93.0 kg) for women. Further, these heavier loads were always paired with one or more gymnastics-calisthenic movements prescribed for multiple repetitions (e.g., 25 toes-to-bar, 50 double-unders, 45–55 handstand push-ups, and 150 feet [45.7 m] of handstand walking). In contrast, the highest load required in all other workouts was for thrusters (men: 100 lbs. [45.4 kg]; women: 65 lbs. [29.5 kg]), and these were typically paired with low-complexity calisthenics (e.g., burpees, burpee box jumps, and double-unders) prescribed at noticeably less volume. The only instance where gymnastics prescription was comparable involved 30 muscle ups (i.e., 12.4–13.3), and the average athlete did not even complete a full round of this workout. While modifying pacing would seem to have a more immediate effect on these latter workouts, improving upon the former would have required more time to develop strength and strength endurance, as well as acquire or improve upon relevant gymnastic skills [27–29]. On average, athletes had less time to work on relevant skills for these three repeated workouts (2.0 years) compared to the other workouts (2.6 years).

An alternate explanation for the observed performance changes involves the relative growth of the sport. Compared to the first CFO in 2011, the number of athletes presently meeting this study’s criteria increased 156–1,305% and 168–1,138% for women and men, respectively. Although the competitive aspect about CrossFit^® has been identified as a highly influential factor for participation and retention [30], little is known about the athletic histories of pre-existing and newly-introduced CFO participants. In a 2016 epidemiological survey of Brazilian CrossFit^® athletes [31], 70.5% of respondents reported being physically active on > 3 days per week doing a variety of activities (i.e., weight training, running, soccer, and martial arts) for several years. Skills learned and developed across various sports and levels of competition are known to have value for an athlete’s primary sport [8, 9], and such skills brought in by newcomers could at least partially explained the improved performances seen in this study. An intentional examination into the development of relevant skills, and whether they were obtained prior to or during CrossFit^® training, would provide greater insight into the factors responsible for the growth of this sport.

The factors responsible for the growth in CFO participation may also be relevant to the differences seen between men and women. In 2011, women accounted for 34.3% of all competitors (or 30.2% of competitors meeting this study’s criteria). Out of all the instances where men outpaced women, half (n = 4) were seen during the repeated workouts of the competition’s first two years. Since 2011, the percentage of competitors who were women increased by 0.8% per year; a 0.3% increase per year out of competitors who met this study’s criteria. Concomitantly, an equal number of instances (initial and repeated) where men (n = 4) or women (n = 4) performed better than the other were seen during this time. No differences were observed in all other instances (n = 7). Moreover, out of the four workouts where men initially performed better, women either eliminated or later outpaced men. Thus, participation in the women’s division is clearly increasing, and women appear to be experiencing greater improvements (~8.3% across all workouts) compared to men (~2.8% across all workouts). The driving force(s) behind these improvements is/are not well understood. It is possible that this observation is a simple mathematical function where lower values do not require dramatic additions in absolute numbers to experience larger percent increases. Alternative explanations include the potential lure of CrossFit^® and competition for women who possess athletic backgrounds [32], social and health (physical and mental) benefits [32, 33], or even a growing realization that this sport empowers women unlike more traditional sports [34, 35].

The presence and adequacy of program scaling are caveats to the observed sex-division differences with pacing. CrossFit^® modifies workout characteristics to presumably acknowledge known physiological differences between biological men and women and present a similar challenge [2, 3, 12]. If applied correctly, performance differences should not exist. However, there were differences in both performance and the application of scaling across exercise types. Resistance training loads, medicine ball weight and target distance, and box jump height were all scaled, but not gymnastic and calisthenic movements (e.g., chest-to-bar pull-ups, handstand push-ups, handstand walking, burpees, rope jumping), continuous exercise patterns (referred to as monostructural, e.g., rowing), or programming durations. Although scaled loads might account for strength differences [13], their arbitrary prescription assumes a specific strength difference that cannot be known across thousands of competitors. Likewise, known differences in body mass [14] might support why gymnastics and calisthenics are not scaled but fails to account for the known upper-body strength differences [13] that are relevant to several of these exercises. Finally, the lack of scaling for continuous exercises or workout durations ignore the typical differences seen with aerobic and anaerobic capacity [13, 15]. Thus, it can be concluded that scaling was not adequately applied, at least across the repeated CFO workouts examined in this study. Others are encouraged to build on these findings by specifically investigating the adequacy of scaled programming assigned to the men’s and women’s CFO divisions.

The present study’s findings are not without limitations. Competition rule and structure changes (e.g., specific days allotted to submit scores, number of workouts, submission card details), as well as variations in when each workout iteration appeared (e.g., the repeated workout 18.4–20.3 appeared on different weeks in 2018 and 2020), and the number of attempts individual athletes made before submitting their best score, have all affected the exact context in which each workout was performed. Nevertheless, these were unavoidable variations that are consistent with the training strategy [4] but should still be kept in mind when interpreting these results. Another limitation involved purposeful, stratified selection of cases based on percent rank to make fair comparisons between sex-divisions and repeated performances. The non-random stratification was done to best approximate a normally-distributed population [19] and since repeated performances were not followed in a random sample of the same exact athletes, it is possible that individual differences and variances in age and relevant physical, physiological (including biological men and women self-selecting to compete in the alternate’s sex division), and psychological traits influenced the analyzed scores [22–25, 36–40]. However, given the growth of CFO participation, it would not have been possible to follow a sufficient sample of the same athletes over a decade of competitions. Even if it were, that sample might be more accurately representative of a specific subset of more experienced CFO athletes, rather than a heterogenous representation of the overall CFO population. Conversely, the opposite rationale underpinned this study’s exclusion criteria. Cases were excluded if the athlete did not earn a score beyond a minimum threshold assigned to each CFO workout of a specific year. These measures were meant to ensure that our findings were representative of a homogenous sample of healthy (i.e., those that did not miss a workout due to injury), well-rounded (i.e., non-workout specialists or those attempting to boost their rank by only completing a small number of repetitions in a specific workout) CrossFit^® athletes. But in doing so, it is possible that representative, low-ranking cases were eliminated, and this could have slightly skewed our results. Finally, the validity of the extracted scores is ultimately reliant on determinations made by CFO competition officials [3]. As these cannot be verified, it is possible that the analyzed scores included errors in reporting, individual variation in meeting exercise movement standards, and outright cheating. Still, the conservative approach in estimating expected sample requirements for sufficient statistical power, and then exceeding the minimum sample size should have assuaged the impact of these limitations on our results.

The findings of this study suggest that performance (measured as repetition completion rate) in most repeated CFO workouts, particularly women, has improved since the competition’s inception. These data serve as initial documentation that, though often scaled, athletes within the men’s and women’s divisions scored differently in ~63.2% of workouts. These differences warrant a more in-depth look across a broader range of workouts, and more specifically, how they might affect acute and long-term physiological responses. Doing so, might help guide more effective, sex-division-equated scaling practices. From a competitive standpoint, the general improvements seen among athletes, along with increased participation, have made it more difficult for athletes to improve their rank relative to their peers. Athletes might maintain their percent-rank but drop in overall ranking even if they complete repeated workouts at a faster rate due to similar improvements made by the average competitor. In this regard, maintaining one’s percent-rank across iterations of a workout should be viewed as a positive outcome, whereas improving upon one’s rank would mean the athlete enhanced their performance to an even greater degree than the remainder of the field. Athletes and coaches are advised to maintain perspective when identifying areas of need, and to focus on finding suitable pacing strategies that balance efficiency with physiological attributes when attempting to improve performance in specific workouts.