2.1.1. Dataset description and collection.

Data for this study were obtained from the 2023 FIFA Women’s World Cup, encompassing all 64 matches played between July 20 and August 20, 2023. The tournament featured 32 national teams, providing a comprehensive dataset of elite women’s football performance at the highest international level.

Our analysis focused on 2,535 non-goalkeeper field player-match observations from athletes who participated in at least one match during the tournament (736 unique players across the full dataset). Goalkeepers were excluded due to fundamentally different performance profiles and goal-scoring expectations. The dataset comprised 51 performance features collected through official FIFA match statistics and categorized into four primary domains:

Physical Performance Metrics (n = 18): Distance covered (total, high-intensity, sprint), speed measurements (maximum, average), acceleration and deceleration events, time spent in different intensity zones.

Technical Performance Indicators (n = 22): Passing statistics (completed, accuracy, progressive), shooting metrics (attempts, on target), ball control measures (touches, successful dribbles), defensive actions (tackles, interceptions, clearances).

Tactical Positioning Data (n = 10): Average field positions, positioning patterns, time spent in different field thirds, spatial movement characteristics.

Methodological Clarification: This is a retrospective analysis using post-match data to identify indicators for future scouting, not real-time match prediction.

2.1.2. Target variable definition and class distribution.

The target variable indicated whether a player scored >= 1 goal in a given player-match observation, creating a binary classification problem with significant class imbalance:

• Goal-scoring observations (Positive Class): 136 observations (5.4%)
• Non-goal-scoring observations (Negative Class): 2,399 observations (94.6%)
• Class Imbalance Ratio: 17.6:1

This extreme imbalance reflects the reality of elite football competition, where goal-scoring opportunities are limited and distributed among a small subset of players.

2.1.3. Data quality assessment and preprocessing.

Exclusion of Match-Outcome Variables: To minimize post-hoc leakage and emphasize individual player actions, we excluded match-outcome-related variables such as TeamScore, Win/Loss, and Game Number. Pre-event team ranking (June 2023, prior to tournament) was retained as a contextual control.

Missing Value Analysis: Initial data quality assessment revealed missing values in 8.3% of observations, primarily due to players with limited match time. Missing value patterns were analyzed to confirm random distribution.

Imputation Strategy:

• Continuous variables: Median imputation within position groups
• Remaining missing values: Overall median imputation after position-level fill
• Preservation of data distribution characteristics

Outlier Detection and Treatment: Numeric features were winsorized to the 5th and 95th percentiles after imputation to limit the influence of extreme values while preserving distribution shape.

Feature Standardization: Standardization (z-score) was applied only within the Logistic Regression and SVM pipelines via StandardScaler inside the cross-validation loop. Tree-based models were trained on unscaled features.

2.2.1. Algorithm selection and rationale.

Five machine learning algorithms were selected to represent different methodological approaches and assess ranking consistency across diverse models:

Tree-Based Ensemble Methods:

• LightGBM: Efficient gradient boosting implementation with fast training [22].
• Random Forest: Bagging with random feature selection for robust predictions.
• XGBoost: Gradient boosting baseline [23].

Linear and Non-linear Methods:

• Logistic Regression: Linear baseline with L2 regularization for interpretability.
• Support Vector Machine: Non-linear classification with RBF kernel for high-dimensional data.

Each algorithm was selected based on proven effectiveness in sports analytics applications and ability to generate probabilistic rankings for Gains Chart evaluation.

2.2.2. Nested cross-validation framework.

To ensure reproducibility and generalizability, we implemented nested cross-validation with the following structure:

Outer Loop (Model Evaluation):

• 5-fold stratified cross-validation
• Maintains class distribution (17.6:1 ratio) in each fold
• Provides unbiased performance estimates

Inner Loop (Hyperparameter Optimization):

• 3-fold stratified cross-validation within each outer training set
• Grid search across algorithm-specific parameter spaces
• F1-score optimization for hyperparameter selection

Two-Stage Metric Selection: Hyperparameters were tuned by F1-score in the inner loop to ensure balanced classification performance; final model selection used Capture Rate @ Top 20% to align with scouting efficiency.

LightGBM Parameters:

• n_estimators: [100, 200]
• max_depth: [16,24]
• learning_rate: [0.05, 0.1]
• class_weight: ‘balanced’

XGBoost Parameters:

• n_estimators: [100, 200]
• max_depth: [16,24]
• learning_rate: [0.05, 0.1]
• scale_pos_weight: automatically calculated as (n_negative/ n_positive) to handle class imbalance

Random Forest Parameters:

• n_estimators: [100, 200]
• max_depth: [1,16]
• min_samples_split: [6,16]
• class_weight: ‘balanced’

Logistic Regression Parameters:

• C: [0.1, 1, 10]
• class_weight: ‘balanced’

SVM Parameters:

• C: [0.1, 1, 10]
• kernel: ‘rbf’
• gamma: ‘scale’
• class_weight: ‘balanced’

This nested structure prevents information leakage between hyperparameter tuning and model evaluation, improving the robustness and reproducibility of performance estimates.

Class Imbalance Handling: Class imbalance (17.6:1 ratio) was handled using cost-sensitive weighting (class_weight = ‘balanced’) rather than synthetic oversampling (e.g., SMOTE, ADASYN [25,26]) to preserve the original data distribution and ranking geometry.

2.2.3. Model selection criteria.

To ensure transparent and reproducible model selection, we pre-specified the following selection criterion before conducting the analysis:

Primary Selection Criterion: Capture Rate @ Top 20% (ranking efficiency)

As motivated in the Introduction, model selection prioritized ranking efficiency. All five candidate algorithms were evaluated under identical nested CV conditions, and the algorithm with the highest Capture Rate @ Top 20% was selected as the primary model (results in Section 3.1). This pre-specification follows best practices for avoiding selection bias in machine learning studies [27].

2.2.4. Permutation test for significance.

To assess whether the observed ranking performance exceeds chance, we conducted a target shuffling (Y-randomization) permutation test. The goal-scoring labels were randomly permuted 100 times while preserving the feature matrix. For each permutation, we trained LightGBM using the best hyperparameters identified from the full hyperparameter search and computed Capture Rate @ Top 20% using 5-fold cross-validation. This produced a null distribution of ranking performance under no signal. The observed capture rate was 0.764, exceeding the null mean of 0.198, yielding p = 0.0099.

2.2.5. Bias mitigation strategies.

Potential biases were mitigated through design choices embedded throughout the Methods. Selection bias was addressed by the nested cross-validation protocol that separates hyperparameter tuning from performance estimation (Section 2.2.2) [27]. Team-dependency and post-hoc leakage were reduced by excluding match-outcome variables during preprocessing (Section 2.1.3). Overfitting and spurious signal were assessed via Y-randomization permutation testing (Section 2.2.4). To reduce analytic flexibility, the primary operational metric (Capture Rate @ Top 20%) was pre-specified for final model selection and emphasized in reporting (Sections 2.2.3 and 2.4). Finally, feature-importance was triangulated across complementary methods (Section 2.3).

2.3.1. Feature importance methodologies.

To ensure robust feature importance estimates, we implemented two complementary approaches:

1. Permutation Importance (Ranking-Based): Model-agnostic method measuring performance decrease when feature values are randomly shuffled [28]. We utilized Average Precision (AP) as the scoring metric instead of accuracy or F1-score. This ensures that the importance reflects the feature’s contribution to the model’s ranking capability, aligning with our primary research objective.
2. SHAP (SHapley Additive exPlanations): Game theory-based approach providing a unified framework for feature attribution [29,30]. This method calculates the marginal contribution of each feature across all possible feature combinations, satisfying efficiency, symmetry, and dummy axioms for theoretically sound importance measures.

Note on Built-in Importance: We excluded built-in Gini importance from the final consensus to avoid potential biases toward high-cardinality features [20] and to maintain a focus on model-agnostic interpretability.

Interpretation Note: Feature importance reflects associations, not causal effects; prospective validation is required before training interventions.

2.3.2. Consensus methodology.

We computed a consensus feature-importance score by averaging, for each feature, normalized Permutation Importance (AP-based; 10 repeats) and mean absolute SHAP values. To summarize stability, permutation importance was repeated using a fixed random seed, and SHAP values were computed on the full analysis dataset with deterministic LightGBM settings.

2.4.1. Metrics.

We evaluate ranking efficiency with gains charts and lift factors that summarize capture rates at fixed review thresholds (see Introduction for motivation).

Lift Factor = (% of goal-scoring observations captured in top k%)/ (k%)

where k is the percentage of top-ranked observations evaluated.

2.4.2. Bootstrap validation protocol.

To quantify uncertainty, we used 1000 bootstrap resamples with replacement; resamples with zero goal-scoring observations were skipped (rare at 5.4% prevalence). For each resample, the model was trained on the bootstrap sample and evaluated on held-out OOB observations to reduce optimistic bias.

Statistical analysis:

• Mean capture rates across bootstrap iterations
• 95% percentile confidence intervals [21]
• Lift factors derived as capture rate/ evaluation percentage

To limit inferential inflation, the primary operational metric (Capture Rate @ Top 20%) was pre-specified, and a single permutation test assessed whether ranking performance exceeded chance. Feature-importance stability was summarized via repeated permutation-importance runs and agreement with SHAP rankings.

Software Implementation: All analyses were conducted using Python 3.13.5 with the following packages: scikit-learn 1.6.1 for machine learning algorithms and cross-validation, LightGBM 4.6.0 and XGBoost 2.1.1 for gradient boosting, SHAP 0.48.0 for Shapley value calculations, pandas 2.2.3 for data manipulation, numpy 2.1.3 for numerical computations, matplotlib 3.10.0 and seaborn 0.13.2 for figure generation. Statistical analyses and bootstrap validation were implemented with reproducible random seeds (seed = 42) for all stochastic procedures. All package versions were pinned in requirements.txt to ensure reproducible benchmarking and stable library compatibility.