https://orcid.org/0000-0001-6641-2225
Figures
Abstract
Sustainability is about meeting the needs of current generations without compromising those of future generations. For fished and aquacultured products, sustainability is often gauged on an individual fishery or farm basis but can be driven by a well-designed and implemented regulatory management system, or set of laws, rules, and policies promulgated by a governmental entity. To achieve global coverage via current assessment approaches requires enormous effort and resources due to the sheer number of individual operating units. System assessments are well established in assuring seafood safety, but much less so in regard to sustainability. This paper presents a tool to measure sustainability based on the prevailing regulatory management system and invites other researchers and practitioners to gauge the applicability of the methodology to new contexts. Evaluating how management systems support sustainable practices is key for assessing current reality and planning change. The methodology presented is based on international criteria for sustainability set forth in United Nations Food and Agriculture Organization (FAO) guidelines for the ecolabelling of fish and fishery products as well as aquaculture certification. The assessment tools are the distillation of qualitative criteria found within the FAO guidelines into simple propositions called “Topics of Pertinence” that enable a straightforward cross-checking of whether the requirements and provisions of regulatory management systems reflect broadly accepted sustainability criteria in fisheries and aquaculture sectors. Assessments document what management systems aim to do via laws, regulations, and policies, and verify mechanisms within the systems that support implementation and compliance. While other sustainability assessments exist, the methodology described here is unique in that it is the only process that provides a system-based approach targeting higher management levels, which can increase global coverage of sustainability assessment from the current lower-tiered approach of certifying at individual production entities.
Author summary
Sustainability certifications generally address individual fishery or aquaculture operations. Only 16% of global fishery and 5% of global aquaculture production have been assessed sustainable by independent third-party organizations. An FAO-based assessment tool for fishery and aquaculture regulatory management systems is presented that applies benchmarking indicators and an evaluation framework that were developed through an FAO expert consultation. The process described enables evaluation of a regulatory management system and its associated governance processes as well as development of options for addressing areas that lack strong evidence of conformance. Assessing management systems is a valid measure of fisheries and aquaculture sustainability. A system approach can increase sustainability assessment coverage of global seafood production.
Citation: Walsh ML, Thorbjørnsen TL, Jones RC (2024) Framework assessment of sustainability: Methodology for evaluating the conformance of fishery and aquaculture management systems to FAO guidelines. PLOS Sustain Transform 3(9): e0000124. https://doi.org/10.1371/journal.pstr.0000124
Editor: Hélder Spínola, University of Madeira, PORTUGAL
Received: September 3, 2023; Accepted: August 15, 2024; Published: September 25, 2024
Copyright: © 2024 Walsh et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: All data are in the manuscript and in publicly accessible documents cited within.
Funding: This research was funded in part by a grant to the College of the Florida Keys in 2016 from the National Oceanic and Atmospheric Administration (NOAA) Sea Grant Aquaculture Research Program (NOAA-OAR-SG-2016-2004807; MW and RJ). The study sponsor (NOAA Sea Grant; https://seagrant.noaa.gov/how-we-work/topics/fisheries-aquaculture/aquaculture/) had no role in the study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: Concurrent with his duties as the Director of Global Aquatic Food Systems at The Nature Conservancy, Robert Jones serves on the Board of Directors of the Global Aquaculture Alliance in an individual capacity. Both Robert Jones and Michelle Walsh have served on the Global Aquaculture Alliance’s Technical Committee to review and modify existing Best Aquaculture Practices (BAP) certification standards and guidelines for Finfish and Crustacean Farms. Thoren Thorbjørnsen has no competing interests.
1. Introduction
Sustainability has become a central component in the production and marketing of fish and seafood products, but the criteria for determining it vary among stakeholders [1]. In its simplest form, “sustainability” is about meeting the needs and wants of current generations without compromising those of the future [2,3]. Current assessments of sustainability, such as those conducted by seafood certification or eco-labeling schemes, primarily focus on the operational management and environmental impacts of a single fishery or farm. These types of assessment processes are viewed by some as inefficient, costly, confusing, and unpredictable especially because the criteria and conditions for approval vary depending upon the scheme [4,5]. Alternatively, one can view sustainability as the result of a well-designed and implemented regulatory management (governance) system, not on a snapshot of an individual operation at any given point in time. A more efficient assessment approach is to focus on the capacity of the governance system to identify and address impacts through management measures that are applicable to all operations under the system’s jurisdiction. A system approach to assessment offers the potential to promote sustainable practices across all fishery and aquaculture operations managed under a national or state regulatory program.
Achieving global coverage via current sustainability assessment approaches requires enormous effort and resources due to the sheer number of individual operating units. Certification schemes have been challenged to achieve significant coverage of global seafood production. After 25 years of operation, approximately 16% of global fishery products have been Marine Stewardship Council certified [6], while only a small proportion of world aquaculture production (5%) has been certified [7,8,9] (although more operations may already be enacting sustainability provisions or have the potential to be certified with some level of improvements). Of relevance to fishery certifications, there are 125 nations that participate in FAO’s Committee on Fisheries [10], twenty-five of which make up 80% of total marine capture fisheries production [11], which is a significantly smaller population to assess than the 2,144 harvested fishery species listed in the FAO global capture database [12]. Aquaculture certifications face similar obstacles in the sheer number of aquaculture operations, which challenges evaluation at a global scale. In the United States alone, the U.S. Department of Agriculture National Agricultural Statistics Service reports that there were 3,456 aquaculture farms in 2018 [13], yet U.S. farms represent less than one percent of global aquaculture production by volume (0.56% in 2017) [11]. In comparison, the FAO maintains data on 107 national aquaculture programs, which may present a more manageable number of units for sustainability assessment [14].
Many certification organizations include an assessment of the management system as one criterion in the certification process. However, current certification programs are primarily based on a “bottom-up” approach to evaluating fisheries or farm sites, and then examine the management structure for those individual operations. Incorporating elements of the “top-down” approach presented here may enable seafood certification schemes to assess a greater share of global seafood production without sacrificing the quality or credibility of these schemes.
Like the Hazard Analysis and Critical Control Points (HACCP) approach to food safety, the assessment methodology presented here takes a system approach to sustainability focused on process rather than an “end product inspection.” The HACCP system is a science-based and systematic process that identifies specific hazards and control points in seafood production to ensure the safety of food. A critical control point is a specific step in a process where control can be applied to prevent, eliminate, or reduce hazards to an acceptable level (e.g., the cooking temperature and duration for raw meat). HACCP is a tool to assess the entire food production process and establish measures that focus on prevention rather than relying mainly on end-product evaluation. Based on risk-assessments of the system, HACCP promotes efficiency by both industry and government in the establishment and audit of safe food production processes. The HACCP method provides flexibility for product control and has been successful in meeting the objectives of ensuring safe seafood acceptable to food control agencies [15].
The tools presented here are designed to evaluate the sustainability of fishery and aquaculture management systems based on criteria set forth by the FAO in two separate documents: Guidelines for Ecolabelling of Fish and Fishery Products from Marine Capture Fisheries (Fisheries Guidelines) [16] and Technical Guidelines on Aquaculture Certification (Aquaculture Guidelines) [17]. The sections on “Minimum Substantive Requirements and Criteria for Ecolabels” (in the Fisheries Guidelines) and “Minimum Substantive Criteria” (in the Aquaculture Guidelines) describe criteria for assessing whether resources of an individual fishery or aquaculture operation are well-managed and sustainable, and are based on international agreements (including FAO’s Code of Conduct for Responsible Fisheries [18]; S1 Box). Both Guidelines concentrate on evaluating discrete operational management techniques implemented on a fishery-by-fishery or farm-by-farm basis.
The assessment tools we present are the distillation of criteria found within the FAO Guidelines into simple propositions called “Topics of Pertinence.” Assessments are based on qualitative data paralleling how criteria have been set by FAO. Our approach considers sustainability criteria in the Guidelines as proxies for baseline sustainable practices and focuses on assessing the standards and procedures of the regulatory management system as a whole. Thus, if a management system ensures that individual operations under its jurisdiction are conducted in conformance with sustainable practices, then the production processes and products are more likely to be sustainable. This type of system assessment would foster regulatory management systems that are proactive and preventative in regard to issues affecting sustainability rather than reactive and recuperative.
This paper develops qualitative tools to enable a straightforward cross-checking of whether the requirements and provisions of regulatory management systems reflect broadly accepted sustainability criteria for fisheries and aquaculture sectors. The structure of this assessment methodology for management systems is unique in that it: (1) is based on a series of FAO Guidelines and an Expert Consultation that provide both globally-reviewed criteria for sustainability as well as a benchmarking framework for assessing conformance, (2) provides a system-based approach to sustainability versus a single-operation approach, and (3) offers a basis to assess sustainability at a higher governance level following a HACCP-like model that can increase global coverage of sustainability assessments from the current lower-tiered approach of individual certified production. This approach is a significant step towards improving management systems by providing a process to document, communicate, and guide the sustainability of fisheries and aquaculture in accordance with criteria published by the FAO.
2. Methods
2.1. Research approach
We developed two qualitative tools based on FAO documents (Table 1) to enable a straightforward cross-checking of whether requirements and provisions of regulatory management systems reflect internationally accepted sustainability criteria for fisheries and aquaculture. The assessment tools are a distillation of the “Minimum Substantive” criteria found within FAO Guidelines into simple propositions called “Topics of Pertinence” (Tables 2 and 3). We worked with intra-agency and multi-stakeholder teams to identify the “Topics of Pertinence” for fisheries and aquaculture, respectively, then to test the tools by applying them to U.S. fisheries and aquaculture management systems (Fig 1). The assessment methodology is based on an FAO evaluation framework for assessing conformance of ecolabeling schemes with FAO Fisheries Guidelines, which describes types of evidence (internal, outcome, independent) that can be documented to verify conformance.
“Topics of Pertinence” of the assessment tools were tracked by highlighting and coding each sustainability provision within FAO Guidelines first by the authors, then discussed, debated, and refined within teams (intra-agency and multi-stakeholder with all authors for fisheries and aquaculture, respectively), then independently reviewed (in the case of the fisheries tool) or agreed upon by consensus (in the case of the aquaculture tool) to ensure validity. The fishery management assessment tool was developed in collaboration with Ocean Trust, which independently applied the tool with state agency teams and review.
2.2. Development of the assessment tools
“Topics of Pertinence” were determined for each distinct sustainability provision identified within “Minimum Substantive” criteria paragraphs of the FAO Guidelines. We phrased each as mutually exclusive as practicable while mirroring FAO verbiage to mitigate bias and minimize inadvertent interpretation of FAO’s intent by the end user. “Topics of Pertinence” were tracked by highlighting and coding each sustainability provision within the FAO documents (Tables 2 and 3); first by the primary author (Walsh), then discussed, debated, and refined within teams (intra-agency and multi-stakeholder with all authors for fisheries and aquaculture, respectively), then independently reviewed (in the case of the fisheries tool) or agreed upon by consensus (in the case of the aquaculture tool) to ensure validity (Fig 1).
2.2.1. Fishery assessment tool.
For the fisheries tool, “Topics of Pertinence” were identified, discussed, and reviewed collaboratively with staff from the National Oceanic and Atmospheric Administration National Marine Fisheries Service (NOAA Fisheries) and Ocean Trust, a former ocean conservation foundation that focused on building science, conservation, and industry partnerships for ocean sustainability. International perspectives on the tool and methodology were provided by members of the NOAA Fisheries Office of International Affairs, FAO representatives, and fishery managers from the international community (including Canada, Mexico, and New Zealand). Then, the fishery assessment tool was peer-reviewed by the Center for Independent Experts (CIE).
2.2.2. Aquaculture assessment tool.
For the aquaculture assessment tool, the authors identified, reviewed, and applied “Topics of Pertinence” collaboratively through a multi-stakeholder steering committee representing diverse aquaculture interests. Eight steering committee members were identified by the authors based on their expertise in aquaculture across the diversity of issues considered in the FAO Guidelines: animal health and welfare, food safety, environmental integrity, and socio-economic aspects. In addition to topical knowledge, the make-up of the steering committee was designed to represent a diversity of perspectives in U.S. aquaculture by including individuals from federal and state agencies, industry, sustainable seafood policy and certification systems, non-governmental organizations, and academia. “Topics of Pertinence” were discussed and debated by the steering committee and authors, and were determined by consensus.
2.3. Assessment steps
For each “Topic of Pertinence” the methodology presents a four-step assessment process to evaluate conformance of a management system to the FAO Guidelines:
- Describe relevant statutes, regulations, policies, guidance, etc. that are applicable to operations under the management system’s jurisdiction.
- Provide evidence of conformance (three types—internal, outcome, and independent; described below in 2.3.1), i.e., how well is the “Topic of Pertinence” being addressed by the management system.
- Rate conformance via a Harvey Ball symbol system (described below in 2.3.2).
- Provide future considerations for how the management system may address gaps or weaknesses in conformance.
The identification of statutes, regulations, policies, guidance, etc. is followed by a discussion section that further describes how these governance mechanisms manage in conformance with the FAO Guidelines, such as providing case studies or clarification in layman’s terms. Documentation of conformance verifies what management systems say they do and tracks those assertions by evaluating evidence as suggested by an FAO Expert Consultation that developed an for assessing the conformity of ecolabelling schemes to the FAO Fisheries Guidelines (Evaluation Framework) [19].
2.3.1. Evidence of conformance.
The Evaluation Framework does not provide specific guidance on how evaluators should make a determination of conformance but it does discuss three types of objective evidence that might be used:
- Internal evidence (i.e. the scheme says it does something);
- Outcome evidence (i.e. the scheme demonstrably does what it says it does); and
- Independent evidence (i.e. an independent expert has determined that the scheme does what it says it does) [19].
For our methodology, internal evidence illustrates how management systems address “Topics of Pertinence” in an official written form. For example, a federal or state constitution, statute, regulation, standard, or strategic plan may state that certain actions will be taken that conform to the FAO Guidelines.
Outcome evidence provides additional support that the management system demonstrably does what it says it does. Examples of outcome evidence include administrative protocols, permitting and reporting procedures that ensure that data are collected and assessments are conducted, evidence that agencies take management action or that laws are enforced, etc.
Independent evidence includes independent expert determinations that the management system does what it says it does. For example, courts have made rulings through the judicial system, open-access evidence that can be confirmed by the general public, or performance that is inspected, reviewed, or verified by an independent or independently-accredited entity or third-party.
2.3.2. Harvey Ball conformance rating.
Conformance of the management system to each “Topic of Pertinence” is rated via a Harvey Ball symbol system for each form of evidence (i.e., internal, outcome, and independent evidence). Harvey ball systems visually communicate qualitative information by assigning round icons to represent the degree to which criteria are being met. Solid symbols (●) indicate conformance to all aspects of the “Topic of Pertinence” and strong evidence for all operations under jurisdiction of the management system, semi-solid symbols (◒) indicate partial conformance with some aspects of the “Topic of Pertinence” and/or variable evidence among operations under jurisdiction of the management system (i.e., strong evidence for some but not for others). An empty symbol (○) in the conformance column indicates a lack of evidence to support strong or partial conformance to the FAO Guidelines. This is not necessarily an indication of nonconformance; it simply demonstrates a lack of evidence available to verify or refute conformance within the management system.
The resultant rating for each “Topic of Pertinence” can be used to identify gaps between the management system and the FAO Guidelines. To lay the foundation for further improvement, a section follows with recommendations for future consideration. In some instances, these future considerations may focus on areas with weaker conformance evidence identified by the process. Suggestions could be provided in a manner that would promote higher conformance on a subsequent assessment. In other instances, particularly for “Topics of Pertinence” in which the management system has scored three solid black symbols (i.e., the highest rating), future considerations may be provided beyond the scope of FAO Guidelines since sustainability may be better envisioned as an ongoing process and not a definitive endpoint. The provided future considerations are not exhaustive, but rather supply potential examples for moving fishery and aquaculture management forward. The most constructive future considerations are those that are applicable, clear, and concrete.
2.4. Testing the assessment tools
In addition to developing the assessment tools, we tested the tools by conducting pilot assessments via within-agency teams and multi-stakeholder steering committees for fisheries and aquaculture, respectively. The assessment tools were applied to the U.S. federal marine fishery management system, to U.S. state marine fishery management systems, and to management systems for aquaculture in the United States.
2.4.1. Assessing U.S. marine fishery management.
From 2012 to 2015 as an agency self-assessment, the first author (Walsh) and colleagues applied the fishery assessment tool to U.S. federal marine fishery management under the auspices of NOAA Fisheries [20]. The self-assessment was reviewed internally within the Office of Sustainable Fisheries, then by representatives of NOAA Regional Offices (West Coast Region, Alaska Region) and by a representative of the Office of Protected Resources. CIE experts were contracted not only to review the assessment tool, but also the NOAA Fisheries self-assessment as well as to generate novel assessments of U.S. federal marine fishery management to further test application of the tool.
Assessments of state fishery management systems were conducted concurrently by the second author (Thorbjørnsen) via Ocean Trust with support from the Gulf States Marine Fisheries Commission. The states assessed were Alabama, Florida, Louisiana, Mississippi, and Texas. Governing laws, regulations, and enforcement actions of each state were assessed, presented to each state management agency for review, and finalized based on state agency comments.
2.4.2. Assessing U.S. aquaculture management.
With support from the NOAA Sea Grant Aquaculture Research Program, the authors coordinated application of the aquaculture assessment tool to the regulatory processes by which aquaculture is managed in the United States [21]. The eight-person steering committee that identified “Topics of Pertinence” for the tool (with minor substitution of members over time) also provided input on the governance and management of U.S. aquaculture and assisted in the identification and evaluation of conformance evidence to the FAO Aquaculture Guidelines.
The steering committee played an integral role in ensuring that necessary and pertinent information regarding U.S. aquaculture management was compiled and available for analysis and rating. Bi-annual meetings of the steering committee were held between November 2016 and August 2019, including two in-person workshops in June 2018 and August 2019. During the August 2019 workshop, the steering committee and authors met to review compiled information on the U.S. management system and assess conformance for internal, outcome, and independent evidence for each “Topic of Pertinence”. The steering committee and authors discussed, debated, and voted on how to rate each “Topic of Pertinence”, and consensus was sought and achieved in each case.
3. Results
3.1. Development of the conformance assessment tools
The fishery assessment tool comprises 24 “Topics of Pertinence” covering considerations for management systems, stocks under consideration, and ecosystem considerations (Table 2). The aquaculture assessment tool comprises 23 “Topics of Pertinence” covering animal health and welfare, food safety, environmental integrity, and socio-economic aspects (Table 3). These subjects mirror those defined in “Minimum Substantive” criteria of FAO Guidelines [16,17]. Application of the assessment approach by those not involved in tool generation (i.e., CIE) verified that the methodology can be applied by independent researchers [22,23,24]. One CIE reviewer stated: “The framework itself is relatively straightforward and closely linked to the FAO Guidelines for Ecolabelling” [22].
“Topic of Pertinence” content is tracked as bolded and underlined text (with respective superscript) within the FAO document excerpts.
“Topic of Pertinence” content is tracked as bolded and underlined text (with respective superscript) within the FAO document excerpt.
3.2. Testing of the assessment tools
The fishery assessment tool was applied by five entities: NOAA Fisheries [20], three CIE experts [22,23,24], and Ocean Trust. The aquaculture assessment tool was applied by one multi-stakeholder steering committee in cooperation with the authors [21]. All assessments indicated high levels of conformance with the FAO Guidelines [20,21,22,23,24].
3.2.1. NOAA Fisheries self-assessment of U.S. federal marine fisheries.
The NOAA Fisheries self-assessment concluded that the U.S. federal marine fishery management system addressed all FAO “Minimum Substantive Requirements and Criteria for Ecolabelling” [20], i.e., all “Topics of Pertinence” indicated evidence of conformance (Table 4). Overall, the percentage of maximum conformance available was very high across all three evidence categories: 97% for internal evidence, 93% for outcome evidence, and 89% for independent evidence (Fig 2). In reviewing the NOAA Fisheries self-assessment, one CIE reviewer stated: “The document contained a very thorough presentation of the extent to which U.S. law and fisheries management meet the requirements of the FAO guidelines [23].
“Topics of Pertinence” are extracted from the FAO Fisheries Guidelines and benchmark indicators of FAO’s Evaluation Framework [19]. Conformance of each “Topic of Pertinence” is described with ● indicating conformance verified by internal evidence, ●● by outcome evidence, and ●●● by independent evidence. Solid symbols (●) indicate strong evidence in all regions and for all fisheries under the management system’s jurisdiction, and semi-solid symbols (◒) indicate conformance with variable evidence among regions or fisheries (i.e., strong evidence in some but not in others). INT = internal evidence; OUT = outcome evidence; IND = independent evidence.
Internal, outcome and independent evidence from Comparative Analysis of U.S. Federal Fishery Management to the FAO Ecolabelling Guidelines: A Self-Assessment [20]. NOAA Fisheries contracted three designees from the Center for Independent Experts to peer review the self-assessment and requested that each CIE Reviewer apply the same methodology to conduct an independent assessment of U.S. federal fisheries management. To consolidate the ratings provided by NOAA Fisheries and CIE assessments, two conservative approaches were applied: Lowest Minimum combined rating is the absolute lowest rating of all assessors combined; Consolidated Numeric combined rating averages ratings across reviewers.
The areas of strongest conformance (denoted by three solid symbols), included attributes of the U.S.’s participatory and science-based fishery management system, specifically:
- complying with local, national and international laws
- developing and abiding by documented management approaches with frameworks at local, national or regional levels
- incorporating uncertainty into stock reference points and catch limits while taking actions if those limits are exceeded
- taking into account the best scientific evidence in determining suitable conservation and management measures with the goal of long-term sustainability
- restoring stocks within reasonable timeframes
- using generic evidence of stock resilience (when necessary)
The areas to be further developed (denoted by semi-solid and/or fewer solid symbols) pertained to the lack of clear, nationwide implementation of ecosystem-based approaches to management, such as incorporating the broad role of the “stock under consideration” in the food-web, and considering long-term changes in productivity.
3.2.2. Center for Independent Experts assessments of U.S. federal marine fisheries.
Through a desk review process, the CIE produced three additional assessments of U.S. federal marine fishery management [20,22,23,24]. For some “Topics of Pertinence,” CIE reviewers rated the performance of U.S. federal fishery management higher than NOAA Fisheries did. For example, all three reviewers felt that there is strong independent evidence available that the U.S. considers types and scales of fisheries in management (Topic of Pertinence #5 of fishery assessment tool), with one CIE reviewer commenting that "Taking into account the types and scales of fisheries in fisheries management does not imply a redistribution of access rights to smaller operators…" [22] and another adding that "Lawsuits in federal and state courts to change allocation between recreational and commercial fisheries" have ensued [23]. All three reviewers also felt that there is sufficient internal evidence by U.S. federal fishery management that stocks are not overfished (Topic of Pertinence #18 of fishery assessment tool).
For some “Topics of Pertinence,” CIE reviewers rated the performance of U.S. federal fishery management lower than NOAA Fisheries did. For example, regarding whether verified traditional, fisher or community knowledge is considered in management, all three reviewers felt there is insufficient independent evidence to that claim. A CIE reviewer noted: "There are no mechanisms for including traditional, fisher or community knowledge in the scientific process. It is an add-on once the scientific advice has been produced and reviewed" [22].
3.2.3. Assessment of U.S. Gulf of Mexico states marine fishery management.
Finalized assessments based on state agency comments were submitted to the respective state agencies (i.e., Alabama, Florida, Louisiana, Mississippi and Texas). Detailed state assessments were not released publicly, but the state assessment process was included in presentations to fishery science conferences and the United Nations Food and Agriculture Organization 2014 Committee on Fisheries 31st Session in Rome. According to Ocean Trust, there was general overall conformance to the FAO Fisheries Guidelines.
2.3.4. Assessment of U.S. aquaculture management.
The assessment indicated that the U.S. aquaculture management system addresses all the FAO’s “Minimum Substantive Criteria” [21]; all “Topics of Pertinence” were supported by evidence of conformance (Table 5). The areas of strongest conformance (i.e., “Topics of Pertinence” given full solid circle ratings unanimously by the steering committee and authors for internal, outcome, and independent evidence) include attributes of the U.S. management system’s participatory and science-based aquaculture management system, specifically:
- compliance with relevant local, national, or international legislation, regulations, and standards
- responsible use of veterinary medicines, chemicals, sanitary measures
- aquaculture facilities located in areas that reduce risk of contamination and potential human health hazards
- aquatic animal feed safety hazards reduced or eliminated
- water quality suitable to produce food safe for human consumption
- identification of environmental impacts
- adverse environmental impacts managed or mitigated
- scientific method used in undertaking risk analysis
- actions taken if environmental impact limits are approached or exceeded
- adoption of efficient water management measures to reduce impacts on surrounding land and water resources
- infrastructure construction and waste disposal conducted responsibly
- recognizing corporate social responsibility from aquaculture to local communities by ensuring fair labor treatment and pay
“Topics of Pertinence” are extracted from the FAO Aquaculture Guidelines. Conformance of each “Topic of Pertinence” is described with ● indicating conformance verified by internal evidence, ●● by outcome evidence, and ●●● by independent evidence. Solid symbols (●) indicate strong evidence in all regions and for all sectors under U.S. management jurisdiction, semi-solid symbols (◒) indicate conformance with variable evidence among regions or sectors (i.e., strong evidence in some but not in others), and empty symbols (○) indicate lack of evidence. INT = internal evidence; OUT = outcome evidence; IND = independent evidence.
The areas to be further developed (i.e., semi-solid or empty symbols) pertain to more explicitly formalizing guidance for minimizing stress on all types of farmed aquatic animals; providing measures to better enable independent verification that workers are trained on good aquatic health and hygiene; proposing incentives to encourage restoration of habitats and sites previously damaged by aquaculture; and establishing policies to internalize the environmental costs of aquaculture operations.
4. Discussion
We designed a system approach to assessing the sustainability of fisheries and aquaculture, which rather than focusing on individual fisheries or aquaculture operations, focuses on evaluating overarching management systems. We based our assessment framework on FAO guidance documents often referred to in the determination of whether fishery and aquaculture operations are sustainable. We tested our tools and methodology by applying them to the U.S. federal marine fishery management system, U.S. state fishery management systems, and to U.S. aquaculture management. All assessments indicated high levels of conformance with criteria set forth by FAO.
4.1. “Topic of Pertinence” approach to tracking FAO criteria
The FAO Guidelines are written in such a way that makes systematic scoring of conformance difficult [20], as noted similarly for the FAO’s Code of Conduct for Responsible Fisheries [25,26]. Therefore, in 2010 the FAO conducted an Expert Consultation to develop an evaluation framework to assess the conformity of ecolabelling schemes with the FAO Fisheries Guidelines [19]. The Evaluation Framework contains a description of “benchmark indicators” that assessors may use to document conformance with the Guidelines. However, benchmark indicators in the Evaluation Framework are not well defined, are duplicative in places, and are not numbered or cataloged, which makes a conformance assessment using these indicators particularly difficult to perform and track.
The conformance assessment tools presented here identify “Topics of Pertinence,” which are key concepts and principles derived directly from the FAO Guidelines and the Evaluation Framework (Tables 2 and 3). Because these FAO documents are internally redundant, similar key concepts and principles reoccur within multiple guidelines or benchmark indicators, respectively. Consolidating key concepts into “Topics of Pertinence” provides an efficient assessment process without losing the substance of the FAO Guidelines. The “Topics of Pertinence” approach thus provides a concise and straightforward assessment strategy against broader language common within United Nations and FAO documents − decreasing the likelihood that evidence for claims of ‘in compliance’ with the FAO Guidelines will be hidden in the details of the overarching text, an issue identified in a FAO briefing document on the utilization of the FAO’s Evaluation Framework for assessing conformance of ecolabeling schemes with FAO Fisheries Guidelines [27]. Like the FAO Evaluation Framework, the current methodology does not weight benchmark indicators.
This conformance assessment methodology is qualitative in nature, as it relies on assessors to identify relevant evidence and rate “Topics of Pertinence.” As a result, the assessment process is susceptible to scoring bias [28,29,30,31]. To mitigate such bias, the phrasing of each “Topic of Pertinence” mirrors the text from the FAO Guidelines and Evaluation Framework (Tables 2 and 3) as closely as practicable to reiterate FAO’s intent, guide assessors, and reduce subjectivity. It is also recommended that assessments and ratings be peer reviewed by independent experts, as we arranged for the NOAA Fisheries self-assessment. Beyond peer review, other strategies can be applied such as employing multiple assessors and merging ratings using a Delphi-method [32,33], which is a structured process for gathering feedback from a panel of experts anonymously through a series of surveys leading to a collective decision.
4.2. Applicability of a system approach to assessing sustainability
The use of system assessments in food production has a long history that created the foundation upon which food safety analyses in the United States and around the world rely. System assessments for food production were conceived in the 1960s when the U.S. National Aeronautics and Space Administration (NASA) asked the Pillsbury Company to design and manufacture the first foods for space flights [15]. NASA’s use of critical control points in engineering management was adapted to identifying and controlling critical control points in food production. The importance of critical control points in food production systems was further highlighted at a 1971 National Conference on Food Protection and used by Pillsbury to organize the first training program on Food Safety through the Hazard Analysis and Critical Control Point (HACCP) System in 1972 [15,34]. The National Academy of Sciences endorsed the use of HACCP compliance audits in place of infrequent and brief food processing site inspections to verify compliance with HACCP criteria in 1982 [15,35]. The application of HACCP to seafood production began shortly thereafter in the early 1980s and led to a global shift in how seafood safety is evaluated: from inspections on the condition of individual end products to a system-based assessment under a HACCP approach.
A system approach to sustainability offers the same potential benefit that seafood safety now enjoys with high compliance across the global seafood production market. The most recent U.S. Food and Drug Administration Evaluation of the Seafood HACCP Program for Fiscal Years 2006–2014 shows that 80 percent of domestic processors are in compliance with HACCP food safety requirements and 81 percent of foreign processors exporting seafood to the United States are in conformance with HACCP requirements [36]. HACCP is a system-based assessment program to ensure food safety that shares the responsibility with seafood producers to be trained in seafood safety criteria and establish HACCP plans. All seafood producers that are HACCP compliant have quality control personnel, many who now also assume responsibility for seafood sustainability. Shifting the responsibility for sustainable seafood production to each individual producer offers the industry an opportunity to be involved in meeting sustainability goals and reduces the inspection burden of certifiers to focus audits on the compliance documents for all producers. Like HACCP, this will require the development and introduction of a training program based on FAO criteria for the development of sustainability plans and implemented globally following the HACCP model.
4.3. Applicability of the current assessment tools and methodology
Incorporating a “top-down” approach to sustainability can enable significant steps to strengthen management systems that have been identified as having gaps in conformance with FAO criteria. In these cases, operations within such management systems could be subject to additional sustainability provisions or conditions, such as more frequent audits, much as in the tiered risk-based approach required under HACCP. In addition to the structured framework that enables evaluation of entire management systems and identification of paths to increased conformance with FAO criteria, additional analyses may be useful to provide greater specificity and clarity needed to address shortcomings (e.g., white papers on specific “Topics of Pertinence” for which there is variable or insufficient conformance evidence). Policymakers, managers, and civil society may identify and consider undertaking a range of actions that can result in full conformance, which could range from less-resource intensive and more feasible actions to major government action that require a broad base of support from its constituency. These may include the development of decision-support tools for use by managers; enhancing, expanding, or creating new management programs or capabilities; reforming or developing new policies or guidance documents; or major changes such as new or revised regulations or legislation.
Our approach is not about competing with traditional ecolabelling or certification schemes: those approaches have merit for the fisheries and aquaculture operations that apply them. Rather, as similarly described for the FAO Evaluation Framework [19], our approach may allow assessment by management systems or third parties that have interest in improving the conformity of a management system with the FAO Guidelines. As described for FAO’s A Checklist for Fisheries Resource Management Issues Seen from the Perspective of the FAO Code of Conduct for Responsible Fisheries [25], the assessment tool may be used “as a focus for discussion by those concerned with management of a given fishery, to be sure that the relevant issues are touched upon” or “for seeing that the fishery in question meets the requirements… which can be updated regularly to see whether progress is being made in approximating the fisheries management system currently in place.” This methodology can be applied at multiple levels of the management system from the national level (or higher), to regional, state/provincial, or local management plan levels. Users may include governments, consumers, retailers, processors or harvesters [19]. We invite other researchers and practitioners to gauge the applicability of the methodology to new contexts, particularly to non-U.S. fisheries and aquaculture management systems such as those of developing world countries with emerging management systems.
5. Conclusion
The assessment methodology presented in this paper provides a means for evaluating management systems to minimum substantive criteria of sustainability put forth by the FAO. This approach is the only comprehensive assessment process based on the FAO Evaluation Framework for evaluating the sustainable performance of management and governance systems while tracking evidence of compliance/conformance that upholds that performance. This process provides a significant tool for assessing management systems to systematically document, communicate, and guide sustainability.
A system approach to sustainability may offer a means to achieve higher conformance to FAO Guidelines in global production. FAO has established an evaluation framework to assess conformance to sustainability criteria that this project refined and tested to show that assessing management systems is an effective tool to evaluate sustainability. Like food safety and the HACCP model, a system approach based on training and producer-developed sustainability plans shares responsibility with the industry that may provide similar comprehensive advancements in seafood sustainability.
Supporting information
S1 Box. Development of FAO Guidelines for fisheries and aquaculture.
https://doi.org/10.1371/journal.pstr.0000124.s001
(DOCX)
Acknowledgments
This work was inspired by recommendations from the 2010 and 2012 Science and Sustainability Forums hosted by Ocean Trust and the American Institute of Fishery Research Biologists, as well as the National Oceanic and Atmospheric Administration’s 2015 pilot application of the methodology in “Comparative Analysis of the U.S. Federal Fishery Management to the FAO Ecolabelling Guidelines: A Self-Assessment” authored by Michelle L. Walsh, Galen R. Tromble, Wesley S. Patrick, and Wendy E. Morrison from the Office of Sustainable Fisheries, NOAA Fisheries. For the fisheries assessment tool, numerous colleagues from within NOAA Fisheries contributed their knowledge and guidance to the formulation and clearance of this assessment. International perspectives on the structure and methodology of the framework were supplied by Greg Schneider, Dean Swanson, Mark Wildman, Ross Shotten, Grimur Valdimarsson, Jorge Csirke, Richard Beamish, and Kevin Sullivan. Additional feedback on our approach was provided by Randy Rice and Steve Cadrin. External peer review of the NOAA Fisheries self-assessment was conducted by Susan Hanna, Cynthia M. Jones, and Jean-Jacques Maguire via the Center for Independent Experts. A dedicated steering committee helped identify and phrase “Topics of Pertinence” for the aquaculture assessment tool as well as apply it to U.S. aquaculture: Benjamin Beck, Robert Bullis, Susan Bunsick, John Hargreaves, Gene Kim, Robert Rheault, Tj Tate, Michael Tlusty, Jonathan van Senten, and Paul Zajicek. Administrative staff at both the College of the Florida Keys (Jonathan Gueverra, Heather Margiotta, Patrick Rice, John Rouge, Brittany Snyder) and The Nature Conservancy (Luann Rudolph, Tiffany Waters) were vital to the flow and completion of the project. We are forever grateful to Christopher Morett of Co|Here Campus and Workplace for his review and editing acumen in making this manuscript more readable to a general audience.
References
- 1. Sainsbury KJ. Review of Guidelines for Ecolabelling of Fish and Products from Capture Fisheries, and Recommended Minimum Substantive Requirements. Report for the Expert Consultation on Ecolabelling Guidelines for Fish and Fishery Products, Rome. 2008 March 3–5.
- 2.
WCED. Our Common Future. Report of the World Commission on Environment and Development, Published as Annex to General Assembly document A/42/427, Development and International Co-operation: Environment. 1987 August 2.
- 3.
United Nations. Report of the World Commission on Environment and Development, General Assembly Resolution 42/187, 1987 December 11.
- 4.
Lassen T, Shotten R, Valdimarsson G, Csirke J. Assessing Coastal State Fishery Management Systems. Terms of Reference. Briefing Note on the FAO Framework Assessment. Ocean Trust. Reston, VA. 2013.
- 5. Ethier V. Certification–Is It Worth The Cost? Aquaculture North America. 2014 December 29 [Cited 2023 September 3]. Available from: https://www.aquaculturenorthamerica.com/certification-is-it-worth-the-cost-1179/
- 6.
Marine Stewardship Council. Celebrating Sustainable Seafood: The Marine Stewardship Council Annual Report 2022–23. 2023. Available from: https://www.msc.org/docs/default-source/default-document-library/about-the-msc/msc-annual-report-2022-2023.pdf
- 7. Bush SR, Belton B, Hall D, Vandergeest P, Murray FJ, Ponte S, et al. Certify Sustainable Aquaculture? Science. 2013; 341: 1067–1068.
- 8.
Aquaculture Stewardship Council. Setting The Standard for Seafood Annual Impacts Report (2022). 2022. Available from: https://asc-aqua.org/wp-content/uploads/2023/08/Annual-Impacts-Report-2022.pdf
- 9.
Global Seafood Alliance. Annual Report 2022. [Cited 2024 July 19]. Available from: https://info.globalseafood.org/2022-annual-report
- 10.
COFI (Committee on Fisheries ‐ Fisheries and Aquaculture Department). Governing and Statutory Bodies Website. Food and Agriculture Organization of the United Nations. Rome, Italy. [Cited 19 July 2024]. Available from: https://www.fao.org/unfao/govbodies/gsbhome/committee-fi/en/
- 11.
FAO. The State of World Fisheries and Aquaculture 2018 ‐ Meeting the sustainable development goals. Food and Agriculture Organization of the United Nations. Rome. Italy. License: CC BY-NC-SA 3.0 IGO. ISBN 978-92-5-130562-1. 2018. Available from: http://www.fao.org/3/i9540en/i9540en.pdf
- 12.
FAO. FAO yearbook. Fishery and Aquaculture Statistics 2017/FAO annuaire. Statistiques des pêches et de l’aquaculture 2017/FAO anuario. Estadísticas de pesca y acuicultura 2017. Rome/Roma. 2019. Available from: http://www.fao.org/fishery/static/Yearbook/YB2017_USBcard/booklet/CA5495T_web.pdf
- 13.
USDA. 2017 Census of Agriculture: 2018 Census of Aquaculture. Volume 3. Special Studies. Part 2 AC-17-SS-2. U.S. Department of Agriculture, National Agriculture Statistics Services. 2019. Available from: https://www.nass.usda.gov/Publications/AgCensus/2017/Online_Resources/Aquaculture/Aqua.pdf
- 14.
FAO. National Aquaculture Sector Overview: Geographic profiles. FAO Fisheries and Aquaculture Department. Food and Agriculture Organization of the United Nations. Rome. Italy. [Cited 2024 July 19]. Available from: https://www.fao.org/fishery/en/naso/search
- 15. Sperber WH, Stier RF. Happy 50th Birthday to HACCP: Retrospective and Prospective. Food Safety Magazine. 2010;15: 44–46. [Cited 2023 September 3]. Available from: http://www.foodsafetymagazine.com/magazine-archive1/december-2009january-2010/happy-50th-birthday-to-haccp-retrospective-and-prospective/
- 16.
FAO. Guidelines for the ecolabelling of fish and fishery products from marine capture fisheries. Food and Agriculture Organization of the United Nations. Rome, Italy. 2009. Available from: http://www.fao.org/3/i1119t/i1119t.pdf
- 17.
FAO. Technical guidelines on aquaculture certification. Food and Agriculture Organization of the United Nations. Rome, Italy. 2011. Available from: http://www.fao.org/3/a-i2296t.pdf
- 18.
FAO. Code of Conduct for Responsible Fisheries. Food and Agriculture Organization of the United Nations. Rome, Italy. 1995. Available from: http://www.fao.org/3/v9878e/V9878E.pdf
- 19.
FAO. Report of the Expert Consultation to develop an FAO evaluation framework to assess the conformity of public and private ecolabelling schemes with the FAO ‘Guidelines for the ecolabelling of fish and fishery products from marine capture fisheries’, FAO Fisheries and Aquaculture Report No. 958. Food and Agriculture Organization of the United Nations. Rome. Italy. 2010. Available from: http://www.fao.org/docrep/013/i2021e/i2021e00.pdf
- 20. Walsh ML, Tromble GR, Patrick WS, Morrison WE. Comparative Analysis of U.S. Federal Fishery Management to the FAO Ecolabelling Guidelines: A Self-Assessment. U.S. Dept. of Commerce, National Oceanic and Atmospheric Administration. NOAA Technical Memorandum NMFS-OSF-1, 2015. Available from: https://repository.library.noaa.gov/view/noaa/9062
- 21.
Walsh, ML, Jones RC, Thorbjørnsen TL. Comparative Analysis of U.S. Aquaculture Management to the FAO Certification Guidelines: An Assessment. 2023 [Cited 2023 September 3]. Available from: https://sites.google.com/view/usaquacultureassessment
- 22.
Maguire J. External Independent Peer Review of the Comparative Analysis of U.S. Federal Fishery Management to the FAO Ecolabelling Guidelines. For the Center for Independent Experts (CIE). CIE Peer Review Reports. Office of Science and Technology, National Marine Fisheries Service, National Oceanic and Atmospheric and Administration, U.S. Department of Commerce. 2014. Available from: https://www.st.nmfs.noaa.gov/Assets/Quality-Assurance/documents/peer-review-reports/2014/2014_10_31%20Maguire%20US-FAO%20ecolabelling%20review%20report.pdf
- 23.
Jones CM. Comparative Analysis of U.S. Federal Fishery Management to the FAO Ecolabelling Guidelines. External Independent Peer Review Prepared for Center for Independent Experts (CIE). CIE Peer Review Reports. Office of Science and Technology, National Marine Fisheries Service, National Oceanic and Atmospheric and Administration, U.S. Department of Commerce. 2014. Available from: https://www.st.nmfs.noaa.gov/Assets/Quality-Assurance/documents/peer-review-reports/2014/2014_10_31%20Jones%20US-FAO%20ecolabelling%20review%20report.pdf
- 24.
Hanna S. Comparative Analysis of U.S. Federal Fishery Management to the FAO Ecolabelling Guidelines Completed for the Center for Independent Experts (CIE). CIE Peer Review Reports. Office of Science and Technology, National Marine Fisheries Service, National Oceanic and Atmospheric and Administration, U.S. Department of Commerce. 2014. Available from: https://www.st.nmfs.noaa.gov/Assets/Quality-Assurance/documents/peer-review-reports/2014/2014_10_31%20Hanna%20US-FAO%20ecolabelling%20review%20report.pdf
- 25. Caddy JF. A Checklist for Fisheries Resource Management Issues Seen from the Perspective of the FAO Code of Conduct for Responsible Fisheries. Series title: FAO Fisheries Circular ‐ C917. 1996. W3140/E. Available from: http://www.fao.org/docrep/w3140e/w3140e00.htm
- 26. Pitcher TJ, Kalikoski D, Pramod G. Evaluations of compliance with the FAO (UN) Code of Conduct for Responsible Fisheries. Executive Summary, Introduction, and Methods. Fisheries Centre Research Reports. 2006;14(2). Available from: https://open.library.ubc.ca/cIRcle/collections/facultyresearchandpublications/52383/items/1.0058147
- 27.
FAO. Evidence on utilization of the FAO Draft Evaluation Framework and the economic impact from ecolabelling on returns to the fisheries sector. Briefing document for Sub-Committee on Fish Trade. Committee on Fisheries. Food and Agriculture Organization of the United Nations. Bergen, Norway. 2014 February 24–28. COFI:FT/XIV/2014/Inf.9 Available from: https://openknowledge.fao.org/handle/20.500.14283/bc431e
- 28. Kahneman D, Slovic P, Tversky A. Judgment under Uncertainty: Heuristics and Biases. Cambridge University Press; 1982.
- 29. Janis I. Groupthink: Psychological Studies of Policy Decisions and Fiascoes. Houghton Mifflin Company; 1983.
- 30. VonWinterfeldt D, Edwards W. Decision Analysis and Behavioral Research. Cambridge University Press; 1986.
- 31. Bell DE, Raiffa H, Tverskey A. Decision Making: Descriptive, Normative, and Prescriptive Interactions. Cambridge University Press; 1988.
- 32. Okoli C, Pawlowski SD. The Delphi method as a research tool: an example, design considerations and applications. Information and Management. 2004;42: 15–29.
- 33. Landeta J. Current validity of the Delphi method in social sciences. Technological Forecasting and Social Change. 2006;73: 467–482.
- 34.
Atkin L, Bauman H, Jezeski J, Silliker J. Prevention of contamination of commercially processed foods. In: Proceedings of the 1971 National Conference on Food Protection. Washington, DC: U.S. Government Printing Office; 1972. pp. 57–83.
- 35.
National Research Council Committee on Food Protection. An Evaluation of the Role of Microbiological Criteria for Foods and Food Ingredients. Washington, DC: National Academy Press; 1985.
- 36.
FDA. FDA’s Evaluation of the Seafood HACCP Program for Fiscal Years 2006–2014. U.S. Food and Drug Administration. College Park, MD. [Cited 2019 August 26]. Available from: https://www.fda.gov/media/130272/download