Sampling of established plants

In 2022, 61 plants were randomly collected from four locations within the Ottawa region of Ontario, Canada: Ottawa Laboratory, Nepean (45.287659, −75.770410), Sandy Hill, Ottawa (45.418558, −75.676766), Experimental Farm, Central Park (45.384524, −75.709887) and Calabogie (45.304705, −76.715290). Ottawa is the capital city of Canada, with an area of 2,778 km², located in the province of Ontario with an area of more than 1 million km². Collection sites were restricted to areas where invasive species are usually present, such as disturbed sites, roadsides, and ditches [34,35]. Collection sites included both public and private land. Sampling on private properties was conducted with authorized access, and no permits were required. Three pictures were taken of most of the collected plants at the collection site to display different parts of the plant (leaf, inflorescence, and whole plant). All pictures were taken with an iPhone 12. Plants were identified to species level through independent morphological testing by trained botanists (the authors AC and AJ) and categorized according to plant family, as well as to distribution status in Ontario: native or introduced. The distribution status was determined using Vascan [36]. In Vascan, taxa that have established following European colonization are labeled as introduced. The collected plants belong to the “established plants” group in the study.

Citizen science records of outsider plants

Plant species from the Canadian Food Inspection Agency (CFIA)’s list of regulated pest plants [37] were selected to represent the “outsider plants” group. These plants are known for their invasive traits and are either not established in Canada or have a very limited distribution. A “collection project” was created on iNaturalist on 2022/09/07 to export user observations of the outsider plants [38]. A set of observations was exported on 2023/01/31 for Echium plantagineum L., Pueraria montana (Lour.) Merr., Microstegium vimineum (Trin.) A. Camus, Aegilops cylindrica Host, and Solanum elaeagnifolium Cav. These species were chosen based on ease of identification and observations were restricted to North America. A randomized dataset of approximately 50 observations for each species were exported. For each observation, the pictures were analyzed by the authors AC and AJ and sorted into “correctly identified”, “incorrectly identified”, and “not possible to identify”, with only the “correctly identified” kept for the taxonomic accuracy assessment. Permission to use pictures with “All Right Reserved” copyright was requested of users, and only those that were granted permission were included in the testing (Table I in S1 Data). In late 2023, a second round of observations was exported to increase statistical power to include Alopecurus myosuroides Hudson, Eriochloa villosa (Thunberg) Kunth, as well additional observations of E. plantagineum and P. montana. New observations were randomly selected from the occurrence data on 2024/01/08 in ‘Beyond a diagnostic tool: Validating standardized Mahalanobis distance as a species distribution model for invasive alien species in North America’ from Stinziano et al. [19] and on 2024/01/09 for E. plantagineum from GBIF [39]. These observations were primarily from iNaturalist in North America, except for A. myosuroides and E. villosa, which include records outside North America due to insufficient data reported at the time in North America. The observations retrieved from Stinziano et al. had already been verified by author AC and sorted according to the same three identification categories noted above. Data for E. plantagineum was exported from GBIF, as this has become the recommended method for using iNaturalist data. Observations of P. montana and E. plantagineum were sorted following the same method. Only the most useful picture for morphological identification was used in the taxonomic accuracy assessment. All pictures included are research grade. The sorting resulted in 402 samples: A. cylindrica (n = 37), A. myosuroides (n = 100), E. plantagineum (n = 44), E. villosa (n = 100), M. vimineum (n = 35), P. montana (n = 46) and S. elaeagnifolium (n = 40).

Taxonomic accuracy assessment

To assess the accuracy of computer vision-based identifications, we conducted a structured assessment focusing on the suggestions provided by the applications from a user perspective, rather than the internal mechanisms of the models. Pictures from the established and outsider groups described above were uploaded to both applications over two trial periods (2022–2023 and 2023–2024). The established group consisted of species with long-established populations, while the outsider group included species not present or with a very limited distribution in Canada. This design enabled assessment of the effect of location precision on identification accuracy, particularly for plants not typically present in a given area. It also allowed comparison of accuracy between native species and long-established introduced species within the established group.

For iNaturalist, an account was created, and pictures were uploaded via the “Upload” page. Selecting the “Species name” field triggered the application’s AI to provide either a single suggestion or a ranked list. Pictures were uploaded to PlantNet via the web interface, also with manual location entry, and the application returned a ranked list of top species suggestions. For both applications, only the first suggestion was recorded, here referred to as “assessment score”. Recording the first suggestion mirrors typical user reliance on the AI recommendations and provides a standardized measure of accuracy. Each suggestion was scored against botanist confirmed identifications using a scale: correctly identified (0), species-level error (1), genus-level error (2), family-level error (3), kingdom-level error (4) or no suggestion-level error (5).

To investigate the effect of location on taxonomic accuracy, location settings were systematically varied, and initial uploads were discarded between tests to prevent carryover effects. In the first trial period, the location in iNaturalist was set to Ottawa (OTT) or Ontario (ON), while the location in PlantNet was set to Canada (CAN) and World Flora, as finer-scale precision was not available. In the second trial period, PlantNet refined its location settings, and “Eastern Canada” was selected to best reflect the collection sites of the established plants. Simultaneously, No Location (NL) was included for iNaturalist alongside the other settings. NL is roughly comparable to World Flora on PlantNet and allowed us to assess AI suggestions in the absence of the location effect. Including this setting also ensured that location effects did not influence other factors being tested, such as differences in accuracy between applications, plant families, and species distribution status.

Due to platform constraints on PlantNet, the outsider group pictures were uploaded exclusively to iNaturalist. For some species (e.g., P. montana), available pictures included only singular leaves, which did not meet PlantNet’s requirements for multiple pictures. Although, PlantNet can provide suggestions from a single picture, limited training data for these species may have prevented this option at the time of testing. Consequently, only data from the second trial period tested on iNaturalist were used to evaluate the effect of location for both plant groups, making PlantNet data irrelevant for this specific research question. Similarly, the data from the first trial period were excluded from analyses related to location, application, plant family, and species distribution status effects as the NL setting was not used.

To examine the influence of plant morphology, a subgroup of 29 samples from the established group was used to test if different parts of a plant (e.g., leaf, inflorescence, whole plant) can impact identification accuracy. Each image was uploaded individually per sample to iNaturalist with no specified location, and the first suggestion was recorded. Pictures were only uploaded during the first trial. For the same reason as the outsider group pictures, they were only uploaded to iNaturalist. Pictures were cropped when necessary to only highlight the relevant plant part under evaluation, due to challenges in achieving precise framing in the field. All pictures from the established group were deposited in a Zenodo repository. Fig 1 provides a summary of the workflow for testing AI-based plant identifications across applications.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 1. Workflow for testing AI-based plant identifications across applications.

The diagram summarizes the stepwise procedure used to assess identification suggestions obtained from iNaturalist and PlantNet across different locations settings and plant parts.

https://doi.org/10.1371/journal.pone.0342712.g001

Statistical analysis

The data collected for statistical analysis included assessment scores from both the outsider and established groups during the second trial period (iNaturalist; n = 463) to test the effect of location. Assessment scores from the established group during the second trial period (iNaturalist and PlantNet; n = 61) were used to test the effects of tool type, family and distribution status. Finally, assessment scores from the established group during the first trial period (iNaturalist; n = 29) were used to test the effect of plant parts. Table 1 provides a summary of the data collected for statistical analysis, the models applied, and the associated predictor variables and interactions, where applicable for each research question.

Download:

• PNG
  larger image
• TIFF
  original image

Table 1. Summary of the models used and relevant parameters to evaluate the taxonomic accuracy score in response to the effects tested.

https://doi.org/10.1371/journal.pone.0342712.t001

The statistical language R, version R 4.3.2 [40] was used to build and run the analysis. The Cumulative Link Mixed Model version 2 (clmm2) from the ordinal package [41], fitted with the adaptive Gauss-Hermite quadrature approximation, was used to evaluate the taxonomic accuracy score, the outcome variable “assessment”. Only the assessment scores 0–3 were used to run the analysis. The Gauss-Hermite quadrature approximation was run with 7 quadrature points. This model was selected as the outcome variable is of ordinal nature and allows the inclusion of random effects and testing of interactions between predictor variables. Random effects (“Lab ID” for the established group and “Catalog ID” for the outsider group) accounted for group-level variability.

The independent variables were entered as fixed factors: location, tool type, family, and distribution status effect. The interaction between the predictor variables, plant family and distribution status with the tool type (iNaturalist and PlantNet) were also included. The pnorm function was used to find the p-values for a right-tail test. The predict function was used to get predicted probabilities for each taxonomic accuracy score. Plotting was performed using ggplot2 [42] to plot the predicted probabilities.

To estimate if the proportional odd assumption (POA) is true, a pairwise comparison of the estimated marginal means between the levels of the predictor variables for each model was computed with emmeans [43]. The emmeans package does not support clmm2. As a result, a cumulative link Mixed Model (clmm) from the ordinal package was used and the output from both models was compared. The results from both models were comparable across all hypotheses, and as such the emmeans package was employed for analysis. When the POA was estimated to be valid, the clmm2 models were retained for use with the predict function, given that the predict function does not support the clmm model. A Proportional Odds Logistic Regression (polr) from the MASS package [44] was used instead of clmm2 when the POA was estimated to be invalid. This model does not support random effects; hence no random effect was included. The brant test [45] was used to test the POA of the polr model. Error rates and standard errors were calculated from assessment scores from the second trial period using Excel to gain insight into the species composition effect within the outsider group according to the location parameters. Overall error rates and standard errors were calculated per species. The following packages were used as complementary tools for plotting, organizing, and analyzing the data: dplyr [46], Hmisc [47], readxl [48], reshape2 [49], tidyr [50] and patchwork [51]. All raw data, scores generated from the analysis and relevant details are presented in the S1 Data.

Location as an input

We assessed if location has a significant effect on the suggestions provided by the computer vision tool and found that there is a strong effect on plants not established in an area (Fig 2, Table 2). We predicted that as the precision of the location input increases, established plants would be identified more accurately, while the misidentification of outsider plants would increase. Our results confirmed only the latter and that the differences between location precision are statistically significant. The established plant group primarily consists of common species that are widespread in and outside Canada (Table D in S1 Data). Common species are often over-represented in the datasets used to train applications. A study by Callaghan et al. [52] found moderate evidence indicating that common bird species were disproportionately represented in iNaturalist. This may explain the lack of difference in identification accuracy between the varying degrees of location preciseness. Furthermore, sampling was conducted in disturbed sites and near roads. Multiple studies have shown that observations are overrepresented in accessible areas, developed areas, managed gardens, or sites near roads or trails [12,53,54]. As a result, computer vision is probably well trained for most of the species in this group. Nonetheless, this result provides direct evidence of a significant bias in iNaturalist, known as spatial bias. This bias causes species that have been observed nearby to be recommended more frequently than those that have not been previously observed in that location. A study conducted by Terry et al. [11] found that geographically constrained identifications tend to be more effective than identifications without such constraints. However, they raised the point that significantly reduced accuracy would result for rarely observed species. Here, we have shown that this is true for species that are scarce in a country where established populations or frequent encounters are lacking. Although we have not tested how sampling bias related to species commonness affects suggestions, we believe this factor likely influences the tool’s performance.

The spatial bias is raising concerns about potential for missed early detections of invasive species. Since the application’s identification recommendations are strongly influenced by spatial data, the outcome can depend largely on how the user records the observation’s information. iNaturalist recognizes that the Geomodel, which uses location data as an input to return suggestions alongside computer vision, is not perfect and are making improvements to enhance the tool’s accuracy [33]. As such, users that rely on the computer vision tool for identification should enter the location parameter after selecting from the suggested species, as then iNaturalist will only use computer vision to provide suggestions based on visual similarity. If the location is inputted beforehand, the Geomodel will return suggestions with the “Expected Nearby” label. Users should take notice and look for the option to include suggestions not expected nearby. An example demonstrating all three cases, with an image of Echium plantagineum from iNaturalist, can be seen in Fig 5. This observation demonstrates how an outsider species might be missed depending on how a user records the observation. It is important to note that the date and location of an observation uploaded to iNaturalist from an iPhone, iPad or android phone should automatically be added depending on the privacy settings or camera GPS capabilities. In contrast, this information is not added automatically when uploading on the web-interface [55]. While this automation is practical, users are encouraged to make decisions on species identification to the best of their knowledge and judgement. Most species suggested after selecting to include suggestions not expected nearby were accurate (AC’s personal observation).

Download:

• PNG
  larger image
• TIFF
  original image

Fig 4. Predicted probabilities of established plants according to taxonomic accuracy and plant parts using iNaturalist.

The difference between pictures with only leaves and with only an inflorescence(s) is significant (p < 0.05).

https://doi.org/10.1371/journal.pone.0342712.g004

Download:

• PNG
  larger image
• TIFF
  original image

Fig 5. Example of iNaturalist suggestions when location is not used and used as an input.

A) Includes suggestions returned when location is not inputted. B) Includes suggestions returned when Ottawa is inputted as the location. C) includes suggestions returned from selecting “Include suggestions not expected nearby” below in B. iNaturalist image of Echium plantagineum was used and we can observe that this species is not among the suggestions until we selected “Include suggestions not expected nearby” when the location is used as an input. Observation 127042112, by brianvanhezewijk [Brian Van Hezewijk], 19/Jul/2022, license is CC BY-NC 4.0. This identification was confirmed by author (AC) as Echium plantagineum with a dried collected specimen sent to the CFIA by Brian in 2024.

https://doi.org/10.1371/journal.pone.0342712.g005

Despite this concern, successful detections remain achievable in iNaturalist, even if they are not the result of AI-assisted identifications. The observation of E. plantagineum in Canada by Brian Van Hezewijk illustrates this perfectly (Fig 5). Brian sent dried flowers from his observation to the CFIA, where the identification was confirmed. Subsequently, the CFIA visited the site and discovered that the small population was no longer present, as the area had been paved over. Although no other plants were found, this type of detection allows monitoring agencies to remain vigilant and investigate possible pathways of entry. The site continues to be monitored by the CFIA. In this instance, the user probably did not select a computer vision suggestion because the small icon used for such identifications is absent. Other success stories exist from using iNaturalist to detect new invasions or distributions [56–59]. For example, Fisher et al. [57] detected a new invasion of the invasive lizard species Aspidoscelis sonorae, which had previously been misidentified as the native Aspidoscelis hyperythrus, by examining photographs labeled as Aspidoscelis sp. in iNaturalist. This case underscores the importance of examining genus-level observations, especially for look-alikes, by scanning citizen-science records to monitor IAP.

Other outcome predictors

Tool type

Based on our findings, we cannot conclusively determine which tool, iNaturalist and PlantNet, provides more accurate identification suggestions. While there is a significant difference between the tools overall, this difference is not statistically significant when considering the interaction of tool type with family and status effect. Although Fig 3 shows slight advantages for each application depending on the plant family or parts, these differences are insufficient to establish which tool is superior for accurate identifications. The small sample size (n = 61) may limit our ability to detect meaningful differences between the tools. As of 2024, PlantNet reports around 7 million users, over 24 million observations and approximately 54,000 species on its statistics page. In comparison, iNaturalist has about 8.2 million users, nearly 87 million plant observations, and about 167,000 species reported on its statistics page or through the Explore query. The greater number of observations and species in iNaturalist suggests it may have an advantage over PlantNet. However, a higher quantity of data does not guarantee better results. Previous research has examined the accuracy of plant species identification across multiple apps, including iNaturalist, Seek by iNaturalist and PlantNet, with many studies showing that PlantNet outperforms iNaturalist or Seek [62,63,67]. While Seek provides suggestions based on data from iNaturalist [68], Campbell et al. [63] found that iNaturalist outperformed Seek, which was among the lowest performing apps. The authors did not provide an explanation for this discrepancy, though we would expect both tools to provide similar suggestions since they are trained on the same data [68]. In this same study, it is important to note that accuracy was assessed based on the “pretty sure” taxon identification, which appears above the list of “top suggestions” in iNaturalist’s computer vision window. However, accuracy should be evaluated on the species listed in the “top suggestions” themselves, as the “pretty sure” result only provides a broad, confident identification at the family or genus level. Although our study’s sample size for this test is small and testing was conducted via the web interfaces rather than smartphones, we believe we have developed a fair and robust method for assessing the performance of both iNaturalist and PlantNet, supported by the use of CLMM modeling.

Simultaneous effects at play

While we have highlighted a significant location effect on the outsider plant group, we must also take into consideration the species composition within this group and its impact on iNaturalist’s suggestions. Four of the species belong to the Poaceae family, which, as we noted with the family effect, may influence the results. As observed in Table 3, the computer vision underperforms the most for species in this family among the outsider group, as fine morphological differences among species characteristics are difficult to capture in photographs. Similarly, E. plantagineum, a species from the outsider group, is distinguished from its lookalike E. vulgare by the number of stamens protruding beyond the corolla [65]. For accurate identification, this feature must be clearly visible in photographs, which can be difficult to capture due to variations in camera angles and lighting. As outlined in Terry et al. [11], picture quality is crucial for the computer vision models to pick up morphological features, and a lack of features may lead the model to rely more heavily on spatiotemporal data. The pictures selected of E. plantagineum for testing on iNaturalist were all verified for this characteristic, confirming the identification. However, the application may lack the training necessary to differentiate between the two species and may rely more on the location effect. Our results suggest this is likely true as the more the location precision increases, the more the error rate increases. To further test this hypothesis, pictures of confirmed E. vulgare with visible stamens and without could be used along with pictures of E. plantagineum in a similar way, with no location inputted.

Additionally, most pictures of P. montana used in testing iNaturalist primarily featured leaves. We noticed that many observations of P. montana on iNaturalist lacked inflorescences or full plant images, likely due to the vine-like nature of the plant. As noted, pictures only including leaves often result in less accurate identifications. The nightshade species, S. elaeagnifolium, is visually striking due to its silver leaves and stems and large purple flowers with yellow anthers, which should yield better identification results. As shown in Table 3, the identification of this species is consistently slightly better than that of the other species, regardless of the location setting within this group. Nevertheless, the location effect still influences the results for this more conspicuous species. Despite the possibility of other effects interacting with the location effect or having their own influence, we demonstrated that when plants are not established in an area, it does affect the suggestions made by iNaturalist.

Limitations, biases and assumptions

Our study is based on several assumptions and limitations. An important underlying assumption is that computer vision is uniformly trained across all species included. While we acknowledged that differences may exist within the species composition of the established and outsider groups, our findings indicate that the location effect remains significant. We also assumed that the level of training for the computer vision is consistent across all taxonomic accuracy assessment testing periods and that picture quality is comparable among samples. However, with iNaturalist releasing a new trained computer vision model each month [33], it becomes challenging to test a large number of samples and different effects. We also assumed that using pre-existing pictures from iNaturalist observations, possibly used to train the model, does not interfere with the identification suggestions made. Additionally, we assumed that the suggestions provided by the tool’s web interface are like those from smartphone apps. In terms of limitations, testing a larger number of established plants is necessary to identify which plant identification tool delivers the greatest accuracy. Despite the small sample size, our findings on family effects and plant parts are consistent with previous research. Furthermore, we occasionally used objects or backgrounds to help bring the plant in focus when taking pictures. This approach was employed to enhance recognition and get a suggestion, especially for fine grass spikelets. We believe that users often utilize similar aids to achieve better outcomes (e.g., hand). Finally, this study addressed the potential impact of three biases on the identification suggestions made by applications, see Table 4. Other biases, related to sampling bias, are certainly influencing the observations reported on these applications.

Download:

• PNG
  larger image
• TIFF
  original image

Table 4. Summary of addressed biases and their effect on the identification suggestions made by applications.

https://doi.org/10.1371/journal.pone.0342712.t004

The sample sizes for each effect tested were 463, 61, 61, and 29 for location, family, distribution status and plant parts, respectively, with tool type included as an interaction for some analyses (Table 1). The smaller sample size associated with some effects were not considered a cause of concern. The taxonomic accuracy scores used in this study are ordinal in nature and therefore comparable to Likert-scale data. Previous analyses comparing parametric and non-parametric testing for such ordinal data have shown that both methods perform comparably when sample size exceed fifteen, with parametric tests generally offering greater discriminatory power among groups [69]. Moreover, assumptions underlying parametric tests do not involve sample size [70], and smaller samples sizes generally reduce statistical power rather than inflate error rates. All sample sizes used here were well above the fifteen-sample threshold. While smaller sample sizes can lead to wider confidence intervals due to greater standard errors, this reflects reduced precision rather than bias and does not comprise the validity of the inferences drawn. Additionally, the assumptions of the cumulative link mixed models applied in this study were tested and found to be satisfied. Finally, the results obtained were consistent and logical, further supporting the robustness of the statistical approach employed (Table 2).