1. Introduction

The role of education in fostering economic growth and social development has been recognized since the origins of modern economic thought. Already in The Wealth of Nations, Adam Smith [1] emphasized that education enhances labour productivity and contributes to the wealth of nations by cultivating skills, discipline, and innovation capacity [2]. In the 20^th century, this idea evolved within the framework of human capital theory [3,4], which conceptualized education as an investment yielding both private and social returns. More recently, the empirical contributions of Abhijit Banerjee, Esther Duflo, and Michael Kremer have demonstrated that education not only drives economic efficiency but also reduces poverty and inequality by enabling more inclusive forms of growth [5,6].

In September 2015, 193 United Nations member states adopted the resolution “Transforming our world: the 2030 Agenda for Sustainable Development” containing 17 Sustainable Development Goals [7]. Within this context, Sustainable Development Goal 4 (SDG 4) “Ensure inclusive and equitable quality education and promote lifelong learning opportunities for all” is an important element of the 2030 Agenda for Sustainable Development. Given the significant differences between countries, including EU Member States, in education levels, quality, and participation in lifelong learning, systematic monitoring and review of SDG 4 implementation is required.

However, dominant approaches to monitoring SDG 4, typically based on composite indicators, face an important limitation. Most aggregation methods are compensatory, whereby high values of some indicators tend to offset the poor performance of others. In the context of SDG 4, this property may lead to misleading assessments, with some countries appearing to perform well overall despite failing to achieve key educational targets, which could lead to unfounded policy conclusions. At the same time, standard composite measures often are characterised by limited interpretability, as they offer rankings without clearly showing the distance from target values or the structure of multidimensional disparities.

The main research goal of the study was to rank the 27 EU countries according to their progress made in 2023 towards achieving SDG 4 goal “Ensure inclusive and equitable quality education and promote lifelong learning opportunities for all” and to determine their distance in relation to the target set for 2030.

To address the limitations of existing approaches, the study proposes a hybrid analytical approach combining multidimensional scaling (MDS) and linear ordering. This framework makes it possible not only to rank countries according to their overall progress toward SDG 4 but also to visualize their relative positions in a two-dimensional space, thereby enhancing the interpretability of multidimensional data. After selecting the optimal multidimensional scaling procedure presented in the methodological part the ranking of objects based on the optimal MDS procedure ensures the stability of the results. By incorporating target values and applying data adjustment, the method provides a more precise assessment of the distance to the target for each EU member state.

2. The characteristics of SDG 4 data

2.1. Objects, indicators, and target levels

The United Nations Goal 4 “Ensure inclusive and equitable quality education and promote lifelong learning opportunities for all” includes seven core targets, which are to be achieved by 2030 [8, pp. 5–6/24]. The global indicator framework contains 231 unique SDG indicators for 17 SDGs (thirteen indicators are used for two or three different targets). SDG 4 included in the UN publication contains 12 indicators [8]. Since 2017, Eurostat has published an “Annual EU SDG indicator review” (https://ec.europa.eu/eurostat/web/sdi/information-data), which contains an updated list of indicators for 17 SDGs (SDGs in the EU context). They are systematically monitored and assessed by Eurostat. Six indicators monitor progress on SDG 4. Four indicators from this list are partially consistent with the UN’s global indicator framework (see Table 1).

Download:

• PNG
  larger image
• TIFF
  original image

Table 1. Indicators and targets for SDG 4 for EU countries.

https://doi.org/10.1371/journal.pone.0333545.t001

In this study, the analysis is intentionally based on the Eurostat SDG 4 indicator set, which ensures transparency, comparability, and policy relevance in the EU context. At the same time, it should be noted that these six indicators primarily capture quantitative and measurable aspects of education systems, such as participation, attainment, and skills, rather than the full conceptual scope of “quality education”. As a result, the study does not aim to provide a comprehensive assessment of educational quality, but rather to evaluate progress made by EU countries in relation to the Eurostat SDG 4 indicators and their associated 2030 target values. This distinction is important for the interpretation of results, as dimensions such as teaching quality, curriculum relevance, or socio-emotional competencies are not directly captured in the available data.

The following analysis is based on average data for the European Union and for 27 EU countries for the year 2023 (the Eurostat database: S1 File). The set of objects included 3 additional ones (S2 File): the target object with coordinates listed in Table 1 (EU 2030 target values), the pattern object (ideal; the upper pole) and the anti-pattern object (anti-ideal; the lower pole). Coordinates of the pattern object represent maximum values for stimulants and minimum values for destimulants (the most favourable values of preference variables), while the coordinates of the anti-pattern object represent minimum values for stimulants and maximum values for destimulants (the least favourable values).

Table 1 presents the indicators used in the analysis along with their target values. All statistical data come from the Eurostat database and are measured on a ratio scale. The selected variables include both stimulants (where higher values are preferred) and destimulants (where lower values are preferred), reflecting different dimensions of educational performance. It should be noted that the interpretation of some indicators requires caution. For example, indicators related to adult participation in learning and digital skills reflect not only educational outcomes but also broader structural characteristics, such as labour market organization, institutional settings, and digital infrastructure. Therefore, cross-country differences may partly reflect these contextual factors rather than education systems alone.

In the case of the variable “Low achieving 15-year-olds in reading, mathematics or science”, the relevant data come from 2022 and are derived from the PISA study (the Programme for International Student Assessment), which is a sample survey conducted every three years by the OECD. Owing to partial data unavailability, information about the variable “Participation in early childhood education” for Greece comes from 2019 and about the variable “Low achieving 15-year-olds in reading, mathematics or science” for Luxembourg – from 2018 (S3 File). These data constraints result from the periodicity and availability of international datasets and should be taken into account when interpreting the results.

Of the six variables shown in Table 1, the variable y3 (Early leavers from education and training) was achieved by as many as 16 countries in relation to the corresponding EU 2030 target of 9% in 2023. The EU target of 45% for the variable y4 (Tertiary educational attainment) was achieved by 13 countries, and the target of 96% for the variable y2 (Participation in early childhood education) was reached by 8 countries. Finland and the Netherlands achieved the target of 80% for the variable y6 (Share of individuals having at least basic digital skills), while Estonia and Ireland achieved the EU target of 15% for the variable y1 (Low achieving 15-year-olds in reading, mathematics or science). Denmark, the Netherlands, and Sweden achieved the target of 26.22% for the variable y5 (Adult participation in learning in the past four weeks).

Two countries managed to achieve four EU 2030 targets in 2023, namely Sweden (for the variables y2, y3, y4 and y5) and the Netherlands (for the variables y3, y4, y5 and y6), where as six countries achieved three targets: Belgium, France, Lithuania, Luxembourg, Poland (for the variables y2, y3 and y4) and Ireland (for the variables y1, y3 and y4). Bulgaria, Cyprus, Germany, Italy and Romania did not achieve any EU targets for 2030 in 2023.

3. Literature review

This section of the study provides an overview of research regarding approaches to measuring progress towards SDG 4. The multidimensional nature of the SDGs – including SDG 4 (Quality Education) – requires the use of different methodological frameworks to adequately assess their implementation. The following review shows that some studies focus on primary data sources and rely mainly on qualitative methods, while others make use of secondary data and apply quantitative methods. At the same time, the literature indicates that the measurement of SDG 4 is not only a technical issue but also involves conceptual challenges related to how “quality”, “equity”, and “lifelong learning” are defined and operationalized in different socio-economic contexts.

The first group of studies includes Ferguson and Roofe [23] and Ferguson et al. [24], who analyse progress towards SDG 4 in high schools, identifying institutional challenges and opportunities for embedding sustainable development in curricula. They employ qualitative analysis, relying on stakeholder interviews and interpretative approaches. These studies stress the importance of subjective, context-rich assessments complementing indicator-based evaluations and argue that educational quality cannot be reduced to quantitative indicators alone, as it depends on institutional practices, teaching methods, and local contexts. Murdoch et al. [25] primarily relying on qualitative assessments, focus on the development and implementation of shared principles supporting the attainment of SDG 4 across European countries, emphasize governance mechanisms and coordination across education systems. Smith et al. [26] integrate both quantitative (logistic regression analysis) and qualitative (document analysis) methods to examine how countries prioritize SDG 4 indicators in the Voluntary National Reviews (VNRs). Their findings suggest that the selection and interpretation of indicators reflect national policy priorities, which introduces an additional layer of heterogeneity in SDG 4 monitoring.

A comprehensive assessment of progress towards achieving SDGs most often requires the use of secondary data sources, which offer a larger number of variables and enable cross-country comparisons. However, in the case of SDG 4, measurement involves specific challenges related to the interpretation of key dimensions. Indicators such as adult participation in learning or digital skills capture not only educational outcomes but also institutional arrangements of labour markets, welfare systems, and digital infrastructure. Similarly, measures of educational attainment or early school leaving may reflect both policy effectiveness and broader socio-economic inequalities. Consequently, the key research problem consists not only in selecting an appropriate method of multidimensional analysis, but also in ensuring that the aggregation procedure does not distort the meaning of these heterogeneous dimensions or lead to misleading policy conclusions. Table 2 provides an overview of studies based on secondary data related to SDG 4 or including SDG 4 as one of the analysed goals.

Download:

• PNG
  larger image
• TIFF
  original image

Table 2. Review of studies based on secondary data related to SDG 4 or including SDG 4 as one of the goals.

https://doi.org/10.1371/journal.pone.0333545.t002

A large group of studies measures all or most SDGs using composite indicators. ASVIS [27] developed composite indicators for all SDGs in the EU using the Adjusted Mazziotta-Pareto Index (AMPI). 88 indicators were synthesized in this approach, including six related to education, to highlight disparities among 28 EU countries. The method enabled the use of multidimensional benchmarking but was limited by the availability and comparability of indicators across member states. Building on this foundation, Miola and Schiltz [18] expanded this perspective using a comparative framework combining the SDG Index, the OECD distance measure, and Eurostat progress metrics. The results of their analysis reveal a strong discrepancy between existing methods and suggest that a country’s relative position depends almost entirely on which method, and which indicators are selected. OECD [11] refined the distance-to-target approach for 36 OECD countries, incorporating nine education-related indicators. The study found that while access to education improved substantially, gaps in learning outcomes and lifelong learning opportunities persisted. The study included time-series data (2005–2017), enabling trend analysis that revealed slow but steady progress in educational equity and participation. Similarly, Hametner and Kostetckaia [28] employed the compound annual growth rate (CAGR) approach to assess the pace of SDG progress. Drawing on 15 years of data, their analysis showed substantial disparities in education performance among EU countries, pointing to differences in policy priorities and investment levels. Bie et al. [29] extended the analysis to Arctic countries, constructing a sustainability index based on the geometric mean of indicators. The inclusion of five SDG 4 indicators revealed that remote regions face unique challenges in ensuring education quality and access, often constrained by geography, demography, and digital infrastructure. Kuc-Czarnecka et al. [30] explored 16 SDGs (excluding SDG 14) using a sensitivity analysis-based approach across 27 EU countries. With five SDG 4 indicators, they examined relationships between education and other goals, identifying strong synergies with SDG 8 (decent work) and SDG 9 (innovation). Qi et al. [31] proposed a Composite Index Analysis Method based on Sachs et al. [32], assessing 193 UN member states across 67 indicators. Their methodological contribution lies in quantifying balance across indicators, providing an alternative lens to distance-from-target approaches by accounting for within-country disparities and uneven development patterns. With two education-related variables, this study highlighted inequalities in educational access and outcomes. Developed countries scored highly on access and completion rates, while developing nations faced persistent challenges in teacher training, resource allocation, and gender parity. Recent work by Blancas and Contreras [33] introduced the Global Sustainable Development Goals Composite Indicator (GSDGCI), which combined compensatory and non-compensatory aggregation rules. By including four SDG 4 indicators, the authors demonstrated how education interacts with other social and economic dimensions, providing a more nuanced understanding of sustainability trade-offs. In parallel, Sachs et al. [34] in the Sustainable Development Report 2025 presented global trends across 167 UN member states showing that despite progress in literacy and school enrollment, SDG 4 remains one of the slower-moving goals due to persistent inequities and pandemic-related disruptions. These studies emphasize that the methodological design of composite indicators—particularly the degree of compensation—has a direct impact on the interpretation of results.

Some studies focused entirely on SDG 4, employing specialized statistical and multi-criteria decision-making methods to explore educational performance and policy implications. Roszkowska and Filipowicz-Chomko [20, 35] developed composite indicators using the TOPSIS and Hellwig methods to measure educational sustainability in 28 EU countries. Using eight Eurostat and OECD indicators, they identified significant disparities between Northern and Western European countries, where education systems performed strongly, and Southern and Eastern Europe, where progress was slower. Their findings underscored the influence of the socio-economic context and policy frameworks on education outcomes. Szymańska and Zalewska [17] employed Principal Component Analysis (PCA) and cluster analysis based on four education indicators. Their results show that EU countries could be grouped into clusters according to their SDG 4 performance, closely mirroring broader socio-economic classifications. The study concluded that education outcomes are strongly correlated with indicators of social inclusion and innovation capacity. In a non-European context, Saini et al. [36] applied Exploratory Data Analysis (EDA) and a genetic algorithm approach to examine India’s progress on SDG 4 between 1990 and 2020. Analyzing 11 indicators, they identified patterns of association between gender parity, literacy rates, and higher education enrollment. Their findings indicate that digital expansion and gender equity significantly enhance education quality, demonstrating the potential of data-driven algorithms for national SDG monitoring. Roszkowska and Wachowicz [16] introduced methodological innovations by combining the entropy weight and Hellwig methods to evaluate education performance across 27 EU countries. Their analysis emphasized how normalization and weighting techniques can influence results, calling for greater transparency in index construction. This contribution is especially valuable for harmonizing methodologies across international SDG assessments.

The above review reveals that measuring progress towards SDG 4 involves both methodological and substantive challenges. On the one hand, composite indicators remain the dominant tool for aggregating multidimensional data. On the other hand, their compensatory nature may obscure structural weaknesses in specific dimensions of education, allowing high performance in one area to offset deficiencies in others. In the context of SDG 4, this issue is particularly important, as balanced progress across multiple dimensions—such as access, quality, and lifelong learning—is a core objective of the goal. As a result, compensatory aggregation may lead to misleading assessments and weaken the policy relevance of rankings.

Against this background, there is a need for methodological approaches that better reflect the target-based and multidimensional nature of SDG 4. The present study responds to this need by applying a hybrid approach combining multidimensional scaling with linear ordering and incorporating target-based data adjustment. This framework reduces the compensation effect and improves interpretability, thereby providing a more reliable basis for assessing progress towards SDG 4 in the European Union.

4. The research methodology

4.1. A hybrid approach to constructing an aggregate measure

A number of different approaches to constructing aggregate measures have been proposed in the literature depending on the degree of compensation [37, p. 3]. Studies on multi-criteria decision-making (see, e.g., [38–40]) uses of non-compensatory methods. An overview of compensatory and partially compensatory aggregate measures applied to different types of data can be found in [41,42].

The following study involves the hybrid method combining multidimensional scaling with linear ordering, which can be used to visualise results of linear ordering for metric data (see Walesiak, Dehnel and Dudek [43]). The procedure of the hybrid method consists of the following steps presented on Fig 1.

The first four steps of the hybrid method procedure include the preparation of the dataset. The hybrid method involves a combination of multidimensional scaling (steps 5–8) and linear ordering (step 9). After applying multidimensional scaling, objects of interest can be visualized in a two-dimensional space. Based on MDS results the objects are linearly ordered according to Hellwig’s [9] aggregate measure using the Euclidean distance from the pattern object.

The characteristics of the steps of the hybrid method in Fig 1 are shown in Table 3.

The method of constructing the aggregate measure by applying the hybrid approach makes it possible to:

• determine the distance of each EU country from the EU 2030 target;
• visualise results of linear ordering of objects in a two-dimensional space. This is not possible when other methods of linear ordering are used;
• enrich the visualisation by including isoquants of development (curves of equal development) and a development path (i.e., the shortest line connecting the development pattern and anti-pattern);
• use the functionality of the mdsOpt R package.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 1. Steps in the procedure of the hybrid method.

Source: own presentation.

https://doi.org/10.1371/journal.pone.0333545.g001

Download:

• PNG
  larger image
• TIFF
  original image

Table 3. Characteristics of the steps of the hybrid method.

https://doi.org/10.1371/journal.pone.0333545.t003

4.2. Selection of the optimal MDS procedure

The Stress measure (the goodness-of-fit statistic) is a summative index and does not inform how well a particular proximity value is represented in the given MDS space (Borg et al. [45], pp. 85–86). When evaluating MDS results we should take into account values of Stress-per-point (spp values) [45, p. 86; 54] and the Shepard diagram [55, 56]. When the distribution of errors associated with the location of individual objects in the scaling space (spp values) deviates considerably from the uniform distribution (e.g., the sum of errors for one or several objects exceeds 40%), the distribution of objects in a two-dimensional space is incorrect. In this case, the ranking of objects based on the results of multidimensional scaling in a two-dimensional space is distorted and does not reflect the actual situation (e.g., objects that should be located closer to the pattern object, are found closer to the anti-pattern object, while other objects may be located above the pattern object or below the anti-pattern object). Problems related to the use of multidimensional scaling in linear ordering are discussed in detail in the paper by Walesiak et al. [43]. The proposed solution in this case involves choosing the optimal multidimensional scaling procedure via application of the mdsOpt package, which offers two criteria: Kruskal’s (standardized residual sum of squares) measure of fit and the Hirschman-Herfindahl HHI index, calculated using Stress-per-point (spp) values. The problem of choosing the optimal scaling procedure (step 5 in Fig 1 and in Table 3) is presented in the paper Walesiak et al. [43].

The optimal MDS procedure was selected by choosing from a combination of nine normalizations methods (implemented in the data.Normalization function from the clusterSim package: n1, n3, n5, n5a, n8, n9, n9a, n11, n12a), four distance measures (manhattan, Euclidean, squared Euclidean, GDM1) and four MDS models (ratio, interval, mspline model – polynomial function of the second and third degree). Fig 2 shows the relationship between and the index, with the best solution denoted by the red circle (procedure 120), which satisfies the condition and minimizes . The Supporting Information section contains results for all MDS procedures. The ranking of objects based on the optimal MDS procedure provides the stability of the results.

Table 4 contains results for the best MDS procedure (120) after applying the hybrid method shown in Fig 1 (step 5).

Fig 3 show a Stress plot and a Shepard diagram for the optimal MDS procedure number 120. The distribution of errors (in terms of stress-per-point) does not deviate considerably from the uniform distribution (when the errors are uniformly distributed, the HHI index is equal to 322.58). The Shepard diagram offers a detailed insight into the goodness of fit between distances in the multidimensional space (Dissimilarities) and the scaling space (Configuration Distances). The diagram presents a general picture of dispersion around the function of regression, making it possible to detect outliers. In the optimal solution (see the Shepard diagram in Fig 3) there are no outliers.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 2. The values of Stress-1 fit measure and the HHI index for MDS procedures (with the best solution denoted by the red circle). Source: The figure produced using the functionality of the package of R [57].

https://doi.org/10.1371/journal.pone.0333545.g002

Download:

• PNG
  larger image
• TIFF
  original image

Fig 3. Stress plot and Shepard Diagram for MDS procedure 120.

Source: own presentation using R [57].

https://doi.org/10.1371/journal.pone.0333545.g003

Download:

• PNG
  larger image
• TIFF
  original image

Table 4. The results for the best MDS procedure (120).

https://doi.org/10.1371/journal.pone.0333545.t004

5. Results in relation to EU-level targets for 2030

Fig 4 shows the results of multidimensional scaling of 31 objects – 27 EU countries, the EU, the 2030 EU target, the pattern object (P) and the anti-pattern object (AP) – representing progress made by EU countries towards achieving SDG 4 in the light of the 2030 Agenda. The use of multidimensional scaling in the hybrid approach allowed for the visualisation in a two-dimensional space the degree to which 27 EU countries managed to achieve the EU 2030 targets for SDG 4 taking into account 6 indicators listed in Table 1. The pattern (P) and the anti-pattern object (AP) were connected with a straight line to form what is known as the set axis. Six isoquants of development divide the set axis into equal parts. EU countries located between each pair of isoquants represent a similar level of progress towards SDG 4. EU countries located at different points on the same isoquant of development have the same level of progress, which results from different combinations of indicator values. The further away from the pattern object of development a given isoquant band is, the less progress EU country located in that band made towards achieving SDG 4.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 4. Visualisation of multidimensional scaling results in a two-dimensional space of progress made by EU countries in 2023 towards SDG 4.

Source: author’s compilation using R [57].

https://doi.org/10.1371/journal.pone.0333545.g004

Taking into account the configuration of objects in a two-dimensional (Fig 4), values of the aggregate measure were calculated according to the aggregate measure . Results of linear ordering of 27 EU countries in terms of their progress in 2023 towards achieving SDG 4 () and the distance of each country from the EU 2030 target () are shown in Table 5. The higher the value of , the more progress a given country made in 2023 towards achieving SDG 4. The higher the value of , the closer a given country is towards achieving SDG 4 in 2030.

The validity of the third approach was confirmed by the application of the procedure described in section 4 for unadjusted data (the second approach). The resulting values of the aggregate measure incorrectly suggest that Sweden, the Netherlands exceeded the 2030 target for SDG 4, while, in fact, neither Sweden nor the Netherlands achieved all 2030 targets for 2030 (see the last paragraph in section 2.1). Sweden did not achieve the targets for “Low achieving 15-year-olds in reading, mathematics or science” and “Share of individuals having at least basic digital skills”, while the Netherlands did not achieve the targets for “Low achieving 15-year-olds in reading, mathematics or science” and “Participation in early childhood education”.

Download:

• PNG
  larger image
• TIFF
  original image

Table 5. Results of linear ordering of EU countries in terms of progress they made in 2023 towards achieving SDG 4.

https://doi.org/10.1371/journal.pone.0333545.t005

The typological classes in Table 5 (last column) were defined in accordance with the mean () and standard deviation () approach (see Table 6). Identical results regarding the typological classes are obtained by substituting values of , and with values of , oraz , respectively.

Download:

• PNG
  larger image
• TIFF
  original image

Table 6. The typological classes according to the level of progress towards SDG 4.

https://doi.org/10.1371/journal.pone.0333545.t006

It is worth noting that in 2023 as many as 13 countries achieved a similar level of progress, represented by the average level of the aggregate measure, ranging from (Croatia) to (Slovenia). This class includes the average EU object.

The distance of particular countries from the EU 2030 target () is visualised in Fig 5. The horizontal line represents the EU-level SDG 4 target value of the aggregate measure .

Download:

• PNG
  larger image
• TIFF
  original image

Fig 5. A graphical representation of distances separating EU countries from the EU target for 2030.

Source: chart created using R [57].

https://doi.org/10.1371/journal.pone.0333545.g005

None of the EU countries achieved this target value although Finland came the closest. The top five countries included the Scandinavian countries, the Netherlands and Estonia (in Fig 5 marked in green). The distance of the European Union as a whole in 2023 in relation to the 2030 target is . Countries that came close to the EU average include Belgium, Portugal, Malta, Czechia and Lithuania. Fourteen EU countries were closer to the target in 2023 than the EU average (see Fig 5). Romania, Bulgaria, Cyprus and Greece were the furthest away from the 2030 target (in Fig 5 marked in red).

To complement the ranking-based interpretation, selected country profiles are presented to illustrate how similar aggregate results can result from different indicator configurations:

• High performers (Finland, Netherlands): Finland, ranked first, achieved the target value for only one indicator (Digital skills). Nonetheless, the country did not fully meet all the other targets of SDG 4, which explains why it still remains below the EU 2030 benchmark. Its profile reflects a balanced and consistently strong performance across different dimensions, in contrast to that of the Netherlands, which ranked second. While it achieved target values for four indicators, including tertiary education and early school leaving, it is still far from the EU targets with regard to one indicator (Low achieving 15-year-olds).
• Middle performers (Poland, Croatia): Poland achieved selected targets (particularly for early childhood education, early school leaving, and higher education). However, it performed significantly worse in terms of participation in adult education and digital skills. Croatia only met the target for one indicator (early school leaving), but is significantly closer to meeting the target for digital skills indicator than Poland.
• Low performers the bottom three countries (Romania, Bulgaria, Cyprus) in the ranking have not achieved any of the EU 2030 targets in 2023. Their poor position reflects persistently low values across a number of indicators, most notably low achieving 15-year-olds and digital skills indicators.

These examples confirm that countries located on the same or adjacent isoquants may reach similar aggregate scores through different combinations of indicator values. Therefore, the results should be interpreted not only in terms of ranking positions but also in terms of underlying indicator structures, which are essential for identifying policy priorities.

6. Analysis of the sensitivity and robustness of the hybrid method

The sensitivity and robustness of the hybrid method was assessed by conducting an additional analysis, in which one or two of the target variables were excluded. The results of linear ordering based on the full and reduced set of variables were assessed using Spearman’s rank correlation coefficient are presented in Tables 7 and 8.

Download:

• PNG
  larger image
• TIFF
  original image

Table 7. The similarity of rankings in terms of values of Spearman’s rank correlation coefficient (sensitivity analysis with the exclusion of one variable).

https://doi.org/10.1371/journal.pone.0333545.t007

Download:

• PNG
  larger image
• TIFF
  original image

Table 8. The similarity of rankings in terms of values of Spearman’s rank correlation coefficient (sensitivity analysis with the exclusion of two variables).

https://doi.org/10.1371/journal.pone.0333545.t008

The results in Table 7 indicate that the application of the hybrid approach is not greatly affected by the exclusion of one variable (values of Spearman’s ranging from 0.8922 to 0.9978). Similarly, the hybrid approach is not much affected the exclusion of two variables, as indicated by the values of Spearman’s ranging from 0.8276 to 0.9869 (Table 8), with the exception of variables y5 and y6, where is considerably lower (0.6268).

The hybrid approach is not seriously affected by missing data in y1 for Luxembourg and in y2 for Greece, as mentioned in subsection 2.1. Even when both variables are excluded from the analysis, is equal to 0.9644 (Table 8).

A sensitivity analysis across normalization choices, distance metrics, and MDS procedures (Supporting information – S1 Table) is a complex problem, which requires a broader discussion and presentation. It turns out that MDS procedures with higher values of the HHI index do not always lead to a considerable change in the ranking of objects. The results may be distorted by the concentration or dispersion of the values of the aggregate measure used to conduct linear ordering. Section 3 of article [43] reports examples of significant ranking changes for MDS procedures with high values of the HHI index. In the hybrid approach linear ordering is preceded by multidimensional scaling in a two-dimensional space. This step causes the loss of information resulting from the shift of the original configuration of objects in a multidimensional space (defined in article by six variables) to a two-dimensional space. Consequently, in the case of these procedures, MDS results in a two-dimensional space must be analysed in terms of the distribution of errors associated with the configuration of particular objects. In the ideal case, the distribution of errors should be uniform (each object should be arranged in a two-dimensional space with the same error). The more the errors of MDS results deviate from the uniform distribution, the worse the results of linear ordering are. This analysis can be facilitated by box plots.

To illustrate the complexity of the problem, the authors analysed ten selected from S1 Table procedures: two procedures of with the best values of the HHI index (no. 120 and 123), and two with the worst values (no. 138 and 137), two with the best values of Stress (no. 2 and 1), two (no. 30 and 99) with the highest and lowest value of Spearman’s with the optimal MDS procedure, and two intermediate ones (no. 100 and 59). Table 9 shows values of Spearman’s between rankings obtained with procedure 120 and the above selected procedures.

Download:

• PNG
  larger image
• TIFF
  original image

Table 9. The similarity of rankings in terms of values of Spearman’s rank correlation coefficient.

https://doi.org/10.1371/journal.pone.0333545.t009

Values of Spearman’s between the ranking based on the best procedure and those obtained by the other selected procedures described in section Supporting information (S1 Table) range from 0.9315 to 0.9980. As can be seen in Table 9, there are no major differences between rankings compared to the one created by the optimal procedure no. 120. This does not mean that all results of linear ordering are equally good. In the case of MDS procedures where the errors are not uniformly distributed, the aggregate measure, used to conduct linear ordering, can be concentrated or dispersed. Table 10 presents the percentage of errors associated with the arrangement of the three worst-performing objects for the selected procedures (with 31 objects analysed in the study, in the case of the uniform distribution, the share of errors for these three objects is equal to 9.677%).

Download:

• PNG
  larger image
• TIFF
  original image

Table 10. The percentage of errors associated with the arrangement of the three worst-performing objects for the selected procedures.

https://doi.org/10.1371/journal.pone.0333545.t010

Major differences in the value of the aggregate measure for the selected procedures are revealed by box plots in Fig 6. The 1^st and 3^rd quartiles for procedure 120 are marked in red, while green values represent the upper and the lower whiskers for the optimal procedure no. 120.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 6. Boxplots for values of the aggregate measure for selected MDS procedures.

Source: chart created using R [57].

https://doi.org/10.1371/journal.pone.0333545.g006

In the case of procedures whose error distributions deviate considerably from the uniform distribution (1, 2, 30, 99, 137, 138) the box plots are flatter than the box plot for the optimal procedure 120. For two procedures whose error distributions deviate considerably from the uniform distribution (59, 100), the plots have longer boxes and longer whiskers, which indicates a greater dispersion of the aggregate measure compared to that for procedure 120. These procedures yield considerably distorted results of linear ordering in terms of the aggregate measure.

Slight differences in the results of linear ordering relative to the optimal procedure were obtained for procedure 123, the second best in terms of the HHI index (normalization method: n11, distance measure: GDM1, scaling model: ratio). The corresponding box plot is not much different from that for procedure 120. Moreover, as can be seen in Table 9, the value of Spearman’s , representing the rank correlation between the ranking created by procedure 120 and that obtained by procedure 123, is very high and equal to 0.9887.

7. Conclusions, discussion, and limitations

Since September 2015, 193 UN countries have been working to implement the resolution “Transforming our world: the 2030 Agenda for Sustainable Development” containing 17 Sustainable Development Goals (SDGs). To ensure sustainable performance in the context of the challenges posed by the 2030 Agenda, each EU country needs to properly assess its progress towards achieving SDGs. Given the importance of this need, we set out to analyse the performance of EU countries in relation to the core targets covered by SDG 4, measured by six indicators. The main aim of the study was to rank the 27 EU countries in terms of their progress towards SDG 4 in 2023 and to determine their distance of individual EU countries towards the 2030 target was also determined. By applying the proposed procedures, it was possible to divide the countries into five relatively homogenous classes with very high, high, middle, weak and very weak levels of progress towards SDG 4.

Supporting information