Training methods and metrics

Ethics

Approval from an ethics committee was not sought because:

1. the data were collected under the auspices of "Captain Elite, Soccer Research and Training Organization", which has no institutional ethics committee;
2. the focal training segment was one component of an ongoing training program and not designed as a specific research study;
3. this study is retrospective and was not anticipated when the data collection took place;
4. all participants and their parents (in the case of minors) signed consent forms that their data could be used for research purposes;
5. all data were anonymized and analyzed anonymously. Training and performance data had no linkages to individual identification information.

Consent was informed and documented via signature. All athletes and parents/guardians signed a waiver/release at every enrollment that included the following statements:

"I understand that the course involves sharing photos, videos and data of Participant during Participant’s training in order to help assess Participant’s level, to share with others in the course and to use for anonymous research purposes. I consent for the Company to use these photos, videos and data for these purposes. If I choose not to participate in sharing data, photos and/or videos of Participant’s progress, I understand that it is my obligation not to enroll in any of the Company’s online training courses. I HAVE CAREFULLY READ THIS RELEASE AND AGREE ON BEHALF OF MY MINOR CHILD."

The collector (M.K.) of the data used in this study is developer and director of a program focused on the training of athletes. This training requires periodic assessment tests to evaluate the rate of development of the athletes (see Skill improvement). These tests are periodically evaluated and results posted so that athletes can see their progress and relative standing. Athletes (and their parents/guardians) who participate in this program sign the above waiver/release specifying that they are aware of the evaluation process and the potential uses of their data, and they are the ones who submit test results and associated video evidence.

Commitment

The program required players to submit a weekly log by midnight every Sunday as a self-report of the work they completed in the training program that week. This log asked players:

1. to record the total number of hours they trained in the training program in the specified week;
2. to specify whether they watched a professional soccer game in the focal week and, If yes, what teams competed;
3. to specify whether they completed each of the tasks required of them for that week (players had to check a box next to each individual task completed for the week).

Each player’s commitment score was based on her/his completion of each weekly task, in addition to whether or not s/he met or exceeded the pre-set goal in weekly hours trained. Additionally, players were given the opportunity to earn extra “points” for their commitment scores by completing optional tasks.

Overall commitment scores were computed as the sum of two components, one based on required tasks and the other based on optional tasks. The “required” component was computed as the fraction of required tasks that was completed; i.e., by dividing the number of points accrued by the maximum number of points for a player completing 100% of required tasks. The required component score thus took values between 0 (no required tasks completed) and 1.00 (all required tasks completed). Optional tasks were available as well, with up to a 1.40 score for an athlete scoring maximum optional points. Thus, commitment scores could range from 0 to 2.40 for each player. The commitment metric was intended to go beyond time expended on training, as it incorporated information that reflected an athlete’s commitment to completing all tasks required of them, as well as some related tasks that were not required.

Hours trained

In the weekly log, players self-reported the number of hours they trained on drills in their specific development tracks. Hours trained were computed for each player by summing the number of training hours reported across the 10-week period.

Skill improvement

The training platform included videos of 10–12 soccer drills that players watched and attempted to replicate on their own. A minimum of three times per 10-week program (beginning, mid-point, and end of the 10 weeks), and a maximum of once per week during the 10 weeks, players were required to self-report their scores on a designated three of these video-based exercises to measure their improvement. Each time players submitted new scores on these 3 drills, they were required to submit videos of themselves completing each drill as evidence that they accomplished the reported scores. Players were encouraged to update their skill scores more frequently, and each time they did so (in excess of the 3 required tests), they were rewarded by gaining more “commitment points”.

Each development track had a different scoring system for measurement of skill improvement. Players measured their skill scores in the dribbling track based on how long it took them to perform each of the 3 designated drills. For the first touch and passing track, they recorded how many correct repetitions they could complete in 30 seconds, and for the other 2 drills they recorded how many consecutive, correctly-executed repetitions they could achieve before making a mistake. For the striking track, players measured all 3 of the drills as the distance from the goal at which they could correctly complete each of the striking techniques. Each striking drill was scored separately as an average of the maximum left-footed and right-footed distances the player achieved. A new distance for either foot on any one of the drills could be achieved only by completing 5 correct strikes in a row with that technique.

A player’s scores from each of the separate development tracks were combined into one overall score using a “Skill Stage” scoring system reflecting objective standards of skill achievement. Scores ranged between 0 and 4 for each development track (0–1.99 = fundamental youth skill, 2–2.99 = elite youth skill, 3–3.99 = collegiate skill, 4+ = professional skill). For this study, skill improvement was measured as a change in average Skill Stage score across each of the 3 designated drills from the beginning of the 10 weeks until the end of the 10 weeks.

Analytic methods

Two sets of analyses were conducted for the 2 different independent variables. The first analysis was based on time spent training over the 10-week training period. The participating athletes showed substantial variation in this metric, ranging from about 16 hours to 90 hours. The second set of analyses was based on the composite metric, commitment, that was computed using a point system, where players were awarded a specific number of points according to which of the required and optional tasks they completed throughout the program (see above).

Training time

Competing models of skill improvement as a function of time spent training were developed and then fit to the data. Model selection was used to assess the level of support for each model. For each of the i = 1…N_A athletes, the primary data used in the analysis were:

y_it = skill level of athlete i at beginning of week t as assessed by skill test,

x_it = time spent training by athlete i between weeks t and t+1.

The focal response variable of the analysis was a statistic reflecting improvement between the initiation of the training program, week t₀, and some endpoint assessment time (in this case 10 weeks later), t_T:

The first independent variable of primary interest was accumulated training time between the beginning and end of the training period (denote the length of the training period as Δt = 10 weeks):

One other independent variable was used in one of the general models for training time, . This variable provided an assessment of skill level at the beginning of the training and assessment period. We did not use as an assessment statistic, because this would induce a sampling covariance between this statistic and our response variable, Δ_i. Use of as an independent variable represented an effort to investigate the hypothesis that initial skill level might influence the increases in skill level expected to accompany increases in training time.

The basic approach of this investigation was to develop 4 models that represented 4 competing hypotheses about the relationship between training time and skill improvement (described above). In all models a normal distribution was assumed with mean μ and variance σ² for the response variable, i.e.,

The models differed only in the structure imposed on the mean, i.e., only in the key determinants of skill improvement. The simplest such model is the null model 1 under which training time beyond 16 hours (the minimum training time among all participants) did not influence skill improvement:

Note that model 1 can be viewed as null with respect to the effect of accumulated hours of training on proficiency. All athletes trained for at least 16 hours during the 10-week program, so μ₁ reflects a baseline level of proficiency that is likely influenced by the first 16 hours of training.

Model 2 was the basic linear model postulating an increase in proficiency with training (prediction: , where the hat denotes an estimate):

Note that no data were available on the effects of the first few hours of training, and rapid increases in proficiency might be expected as drills are learned. Thus, models 2–4 should be appropriate for predicting changes in proficiency for ≥16 hours of training.

Model 3 included an intercept term, β₀, a linear term, β₁, describing the relationship between training time and proficiency, and a quadratic term, β₂, accounting for a possible threshold point after which time spent training begins to have either an increased or diminished effect:

The prediction was that (where hats ^ denote estimates), indicating an increase in skill improvement with increased training time; and , indicative of greater difficulty in increasing proficiency as training time and proficiency increase.

Finally, another general model, 4, included a linear term describing the relationship between training time and performance, as well as an interaction term expressing the additional influence of starting proficiency level:

The above model simply states that the average skill improvement can be written as a linear function of some overall baseline effect, β₀, an effect of additional time spent training, β₁, and an interaction effect between initial skill level and time spent training, β₄. The expectation was that , indicating an increase in skill improvement with increased training time; and , indicating a reduced effect of training for athletes with high initial skill levels. Note that unlike models 1–3, each of which postulates a single relationship for all individuals in the population, model 4 produces a different relationship for individuals with different values for , yielding a family of relationships associated with the different starting proficiencies.

Each of these 4 models was fit to the data, and model selection statistics and parameter estimates were then used to judge which model “best” described the data. Models were fit using maximum likelihood, and Akaike’s Information Criterion (AIC) [14, 15] was used for model selection. Based on the assumed Gaussian model, the maximum likelihood estimator for each of the models 1–4 takes the form of the regression equation (1)

The resulting parameter estimates provide information about the importance of each independent variable. Each effect estimate, , was examined to determine whether its sampling distribution, the width of which is reflected by the associated standard error (see S1 Appendix), , was centered near 0 (indicating little support for the effect) or far from 0 (supporting the effect).

Because all 4 models represented plausible hypotheses, and because no single model received overwhelming support relative to the other models, model-averaged estimates [15, 16] were computed for the β parameters that defined our different models (see S1 Appendix).

Commitment.

It has been suggested that perhaps a better predictor of performance improvement is “grit” (popularized in [12]), a term intended to reflect not just the time spent training for an activity, but the quality of that training and the dedication of the individual athlete to it. This grit characteristic is referred to here as commitment and is quantified over the 10-week training period as described above. As with training time, an increase in improvement was predicted for those athletes with higher commitment scores. A stronger relationship was predicted for commitment than for training time, but because the scales of these 2 independent variables are not comparable, this expectation does not lead to predictions about relative magnitudes of the effect estimates, . Both training time and commitment could not be included as independent variables in the same model(s), because training time is a component of the commitment measure and these variables were relatively highly correlated. In order to avoid potentially problematic issues with collinearity, model selection was used to compare the models developed for these 2 independent variables.

The basic modeling approach described above for time spent training was also used for the commitment variable, z_it.

The commitment variable corresponding to the end of the 10-week assessment period was denoted as z_iΔt for consistency with the time variable. The same basic set of models as used for time spent training was used to assess the relationship between accumulated commitment and improvement in proficiency. Thus, z_iΔt was substituted for x_iΔt in all of the above expressions in order to conduct this second set of analyses.

Training time

Model selection statistics for time spent training are presented in Table 1. The “best” model (ΔAIC = 0) was model 2, in which there was a linear increase in proficiency with training time. Model 3 was very competitive, showing evidence of an increase in proficiency with training time until about 80 hours, after which additional training did not appear to further increase proficiency (Fig 1). Model 4 received less support than models 2 and 3, but provided some evidence of greater increases in proficiency for athletes beginning training at lower proficiency levels. The lowest ranked model was model 1 (no improvement with training time). Estimated β parameters describing these relationships (Table 1) provided strong evidence of an increase in proficiency with increased hours of training time.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1.

(left) Data and corresponding model weights for each of the 4 models described in the text. The model 4 plots are of the extrema coinciding with the smallest and largest interaction effects (the lowest and highest beginning skill levels), so that there are 2 curves associated with this model. (right) Weighted average model results obtained using AIC weights. All models predict improvement with training, however the exact nature of the relationship between improvement and training time is less clear. The model-averaged predictions (right) represent our best assessment of overall training effects, given this model uncertainty.

https://doi.org/10.1371/journal.pone.0276762.g001

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Model parameter estimates and standard errors, along with the associated ΔAIC values indicating the relative support for each of the four models using “training time” as the independent variable.

https://doi.org/10.1371/journal.pone.0276762.t001

The raw data for each athlete in the training program, as well as the relationships between proficiency and training under all 4 models, are plotted in Fig 1. The raw data show substantial variation, emphasizing the need for statistical inference in order to draw conclusions about the effectiveness of training. Also shown in Fig 1 is the model-averaged relationship, which can be viewed as providing the best assessment of overall training effects, given model uncertainty. The model-averaged relationship shows increases in proficiency with hours of training, consistent with a priori hypotheses and the purpose of the training program.

Commitment

Following the same modeling and estimation approach as above, the analyses were repeated with “commitment” score as the independent variable. The quadratic (model 3) and linear (model 2) models were both competitive, showing increases in skill improvement with larger commitment scores (Table 2, Fig 2). Given model uncertainty, the model-averaged plot in Fig 2 provides the best description of this relationship.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2.

(left) Data and corresponding model weights for each of the 4 models described in the text, but with z (commitment) as the independent variable. For model 4, the extrema coinciding with the smallest and largest interaction effects are again plotted. (right) Weighted average model results obtained using AIC weights. The model averaged predictions (right) again represent the best assessment of overall training effects given the model set considered.

https://doi.org/10.1371/journal.pone.0276762.g002

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Model parameter estimates and standard errors, along with the associated ΔAIC values when using commitment as the independent variable.

https://doi.org/10.1371/journal.pone.0276762.t002

Combined analysis for training time and commitment

To address the question of which independent variable provided a better description of the data, the training time and commitment models were combined to form a single model set with new model weights. In this “joint” assessment, seven separate models are considered, and their AIC values are labeled as AIC_CT1, AIC_C2, AIC_C3, AIC_C4, AIC_T2, AIC_T3, AIC_T4, where the subscripts denote the independent variable (T = time, C = commitment), and the form of the model (1–4) for that independent variable. The first model (constant) is identical for both independent variables (time, commitment); hence only the single AIC_CT1 value need be considered.

The model weights corresponding to use of commitment as the independent variable are all considerably larger than those using training time as the independent variable (Table 3). This suggests that commitment is a better predictor of performance improvement than time spent training. The largest model weight corresponds to the model that predicts a quadratic relationship, showing an increase in performance with commitment score that levels off at higher levels of commitment. An evidence ratio was computed using the AIC weights of the best models for each covariate, models C3 and T2. The evidence ratio, w_G3 /w_T2 = 5799, strongly supported the quadratic commitment model as a better model for performance improvement than the linear training time model.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. AIC, ΔAIC, and model weights obtained by considering the joint model set consisting of 6 models associated with using commitment (C2-C4) and training time (T2-T4) as the independent variables, as well as a constant (null) model, CT1.

The “constant” model is the same for the 2 independent variables.

https://doi.org/10.1371/journal.pone.0276762.t003