Fig 1.
The PHOTONAI framework is built to accelerate and simplify the design of machine learning models. It adds an abstraction layer to existing solutions and is thereby able to simplify, structure, and automate the training, optimization, and testing workflow. Importantly, the pipeline and the workflow are subject to user choices as the user selects a sequence of processing and learning algorithms and parameterizes the optimization and validation workflow. The here depicted class diagram shows PHOTONAI’s core structure. The central element is the Hyperpipe class, short for hyperparameter optimization pipeline, which manages a pipeline and the associated training, optimization, and testing workflow. The Pipeline streams data through a sequence of n PipelineElements. PHOTONAI relies on the established scikit-learn [7] object API, to integrate established or custom algorithms (BaseElements) into the workflow. PipelineElements can have n hyperparameters which are subject to optimization by a hyperparameter optimization strategy.
Fig 2.
A: The Switch class represents an OR-Operation and can be placed anywhere in the sequence to interchange and compare different algorithms at the same pipeline position. B: The Stack represents an AND-Operation and contains several algorithms to share a particular pipeline position. It streams the data to each element and horizontally concatenates the respective outputs (see Listing 3). Next to generating new feature matrices through several processing steps at runtime or building classifier ensembles, it can, in addition, be used in combination with the branch element.
Fig 3.
Example plots of PHOTONAI’s result visualization tool called Explorer. A: User-defined performance metrics, here accuracy, precision and recall, for both training (blue) and test (dark) set. The horizontal line indicates a baseline performance stemming from a simple heuristic. B: For regression problems, true and predicted values are visualized in a scatter plot on both train (left) and test (right) set. The values are generated by the best model found in each outer folds, respectively. C: Hyperparameter optimization progress is depicted over time for each outer fold. D: Pipeline elements and their arrangement is visualized including the best hyperparameter value of each item.
Table 1.
Validation performance metrics for three different pipeline setups.
Table 2.
Different estimator’s average best validation performance for the class balancing pipeline.
Table 3.
Test performance metrics for the final model.
Fig 4.
Parallel plot showing hyperparameter exploration.
All three hyperparameters used in the final model are shown on the x-axis. They include the class balancing algorithm and two hyperparameters of the Random Forest classifier (maximum number of features and minimum samples per split). Each line represents a specific combination of all hyperparameters. The line color reflects the corresponding model performance based on the f1 score. Higher model performance is shown in dark red while lower model performance is shown in blue. Random under-sampling appears to increase model performance slightly while using no class balancing algorithm decreases overall model performance. CB = class balancing, RUS = random under-sampling, ROS = random over-sampling, SMOTE = synthetic minority oversampling technique, RF = Random Forest.