Governing partnerships with technology companies as part of the COVID-19 response in Canada: A qualitative case study

Cross-sector partnerships are vital for maintaining resilient health systems; however, few studies have sought to empirically assess the barriers and enablers of effective and responsible partnerships during public health emergencies. Through a qualitative, multiple case study, we analyzed 210 documents and conducted 26 interviews with stakeholders in three real-world partnerships between Canadian health organizations and private technology startups during the COVID-19 pandemic. The three partnerships involved: 1) deploying a virtual care platform to care for COVID-19 patients at one hospital, 2) deploying a secure messaging platform for physicians at another hospital, and 3) using data science to support a public health organization. Our results demonstrate that a public health emergency created time and resource pressures throughout a partnership. Given these constraints, early and sustained alignment on the core problem was critical for success. Moreover, governance processes designed for normal operations, such as procurement, were triaged and streamlined. Social learning, or the process of learning from observing others, offset some time and resource pressures. Social learning took many forms ranging from informal conversations between individuals at peer organisations (e.g., hospital chief information officers) to standing meetings at the local university’s city-wide COVID-19 response table. We also found that startups’ flexibility and understanding of the local context enabled them to play a highly valuable role in emergency response. However, pandemic fueled “hypergrowth” created risks for startups, such as introducing opportunities for deviation away from their core value proposition. Finally, we found each partnership navigated intense workloads, burnout, and personnel turnover through the pandemic. Strong partnerships required healthy, motivated teams. Visibility into and engagement in partnership governance, belief in partnership impact, and strong emotional intelligence in managers promoted team well-being. Taken together, these findings can help to bridge the theory-to-practice gap and guide effective cross-sector partnerships during public health emergencies.


Introduction
COVID-19 has upended health systems, economies, and daily life around the globe. Given the immense burden the pandemic has placed on health systems, there has been a resurgence in the discussion of health systems resilience among practitioners and researchers [1]. Health systems resilience reflects preparedness for an emergency as well as the capacity in the system to respond effectively to emergencies while maintaining core functions [2]. Health emergencies such as pandemics have broad impacts across society and thus a key principle that characterizes resilient health systems is the capacity to coordinate actions across the public, private, and non-profit sectors [2].
Past public health crises such as pandemic H1N1 and the 2014-2016 Ebola epidemic have generated calls to action around fostering effective cross-sector partnerships [3][4][5]. The COVID-19 crisis is unique due to the previously unprecedented role of digital technologies, and thus private technology companies, in pandemic response [6]. From digital contact tracing (DCT), to ledgers organizing vaccine administration, to machine learning-based risk prediction models of severe clinical outcomes-medical and public health organizations have sought to leverage digital technologies to support pandemic response [7][8][9]. However, health organizations traditionally do not have the skillset to develop and deploy such technologies in-house, especially in the context of a rapidly evolving public health emergency [10]. Therefore, Using a qualitative, multiple case study design, we sought to study the governance arrangements between Canadian health organizations and private technology startups during the COVID-19 pandemic. Our objective was to systematically apply a framework of adaptive governance in healthcare to describe and analyze the barriers and enablers of deploying digital technologies as part of the pandemic response. The results of this work are vital to help bridge the theory-to-practice gap and to guide effective and responsible cross-sector partnerships during public health emergencies [42].

Methods
This qualitative study was conducted through use of a multiple case study design [43]. Using a case study design allows researchers to illuminate why a decision-or set of decisions-were taken, how they were implemented, and with what results. When describing case-study research in his popular methodological manual, Robert K. Yin puts forth that a case study "relies on multiple sources of evidence, with data needing to converge in a triangulating fashion" [43]. Examples of common evidence sources researchers select to conduct in-depth study of a phenomenon in its real-world context include: documentation, archival records, interviews, direct observations, participant-observation, and physical artifacts [43]. We selected the complementary methods of a document review as well as interviews with key informants. The purpose of the document review was to provide context into the need for study partnerships, provide a starting point for identifying potential key informants, and gain an understanding into key events and outcomes of the partnerships. Key informant interviews were conducted to explore a range of topics from the drawn upon theoretical frameworks (i.e., Vayena and Blasimme's framework of systemic oversight, Gasser's navigation aid for the ethical use of digital public health tools) that we believed would not likely be captured in documents. Key informant interviews can elucidate why and how certain decisions were made-that is provide critical perspectives into partnership governance. Interviews also allow for researchers to challenge their participants by probing on what they thought could have gone differently or be improved upon. Ultimately, we believed that document review and the key informant interviews would provide us with complementary and synergistic findings and contribute to a robust case study. We explain the methodological basis for how we conducted both below.
Our cases were sampled from three separate public/private collaborations to deploy digital technologies as part of the COVID-19 response. These collaborations were broken down into three categories of technologies as outlined by Shaw et al. (2019): clinical, operational, and epidemiological [44]. While there are other taxonomies that have been applied to digital technologies in the context of the pandemic [20,21], these taxonomies and their subcategories are not mutually exclusive. Each of the technology companies were startups.
Technology company A and health organization B partnered to deploy a virtual care platform for staff at long-term care homes to consult physicians regarding the status of COVID-19 patients without having to send them to an emergency room. This virtual care platform also facilitated COVID-19 testing for patients within health organization's catchment area. Shaw et al. (2019) defined "clinical applications" as those pertaining to the tasks of assessing, evaluating, and intervening on health issues [44]. Since these applications are directly related to the task of patient assessment and evaluation, we selected this partnership as our "clinical" collaboration.
Technology company C and health organization D partnered to deploy a secure messaging platform for physicians. Physicians used this platform, in part, to coordinate the care of COVID-19 patients. Physicians would receive notifications on this platform when tests returned positive with SARS-CoV-2, use the platform to request consults with other services, and serve as a digital pager when on call. As per the definition proposed by Shaw et al. (2019), these tasks are ancillary to, but essential for delivering clinical services [44]. Therefore, we selected this partnership as our "operational" collaboration.
Technology company E partnered with health organization F to provide data science consulting regarding the management and spread of COVID-19 at the population-level. Part of this consulting involved leveraging novel, digital data streams. As such, we selected this partnership as our "epidemiological" collaboration of study in line with the definition proposed by Shaw et al. (2019) [44].
The methods were performed in accordance with relevant guidelines and regulations and approved by the University of Toronto's Health Sciences Research Ethics Board (REB #26448).

Theoretical framework
Since we are studying the governance of collaborations between Canadian public health/health care organizations and private technology companies, we have drawn from a governance framework to inform the methodologies and analysis of our study. Specifically, we used Vayena and Blassime's framework of systemic oversight [45][46][47].
Systemic oversight draws upon the extensive literature on adaptive governance, originally developed for application in the environmental sciences. Adaptive governance is a means of coordinating resource management within uncertainty-laden and rapidly changing socio-ecological systems [48,49]. Vayena and Blassime argue that health data, due to its sources, capabilities, and stakeholders, can be characterized as an "ecosystem", presenting significant implications for its management [45]. The framework is aimed at promoting responsible innovation across biomedical research, patient care, and public health with three aims: 1) to draw attention to ethically controversial areas to address risks, 2) to direct innovation towards outcomes beneficial for society, and 3) to promote public trust and accountability [46]. In keeping with these objectives, systemic oversight: 1) leverages adaptive and flexible mechanisms to cope with uncertainty, 2) is both inclusive and reflexive to engage a wide range of stakeholders and consider how they may be impacted by a given solution, and 3) monitors for and is responsive to emerging risks and harms [45][46][47].
There are numerous conceptual strengths of this framework for our study. First, it recognizes the ecosystemic and highly interconnected nature of both the digital technologies and the institutions involved in COVID-19 response. Second, it is appropriate for understanding how organizations respond to the COVID-19 pandemic, which is inherently a situation in which actors must act under tremendous uncertainty, time pressures, and constant change (e.g., in the epidemiological and clinical dynamics of the disease, spillover impacts to other aspects of society such as supply chains, etc.). Adaptive governance has been used in similar areas such as managing and mitigating risk due to natural disasters [50]. Third, adaptive governance also aptly relates to the notion of resiliency as both draw from the environmental sciences [48]. Finally, the use of this framework lends itself towards a hypothesis that is grounded in what has been seen in adjacent literature: when the structure of an ecosystem is challenged by crisis, then the mechanisms of governance that were designed optimally to support the ecosystem are at the heart of the crisis and must adapt in ways for which they were not designed.
We opted to employ Blasimme and Vayena's framework of systemic oversight over other frameworks such as Char's pipeline [51] or Leslie's process-based governance framework for AI workflows [52] due to its complementarity and specificity with this study (i.e., we are focused on the governance of a health response to a crisis rather than the ethics surrounding a response). We also focus on the use of digital technologies to meet this response rather than artificial intelligence tools, strictly speaking. We also drew from Gasser et al.'s navigation aid to guide our data collection through semi-structured interviews with key informants [20]. We structured our interviews in accordance with the aid's conceptual phases of developing and deploying digital technologies: 1) Preparation, 2) Planning, 3) Risk Assessment, 4) Development, 5) Deployment and Evaluation.

Document review
Document analysis included documents that provided details on the structure and nature of the sampled partnerships. Documents were included if they included details of the partnership, leadership perspectives, outcome measures, regulatory landscape, or product features. The search for these documents was done through multiple Google searches and manual searches in sampled organization webpages by following links. This process was done independently by two individuals (VH, TS) while keeping track of search terms (Appendix A in S1 Text). Searching and data abstraction was done by two independent reviewers (VH and TS), with conflicts resolved by discussion. We performed two rounds of document review (summer 2021, winter 2022) to account for new updates regarding the partnerships or organizations involved during the study period.

Key informant interviews
An initial list of key informants was first identified through the document analysis. We then used snowball sampling to identify additional participants at the end of each interview. Participants were recruited by email and were sent a standardized consent form (Appendix B in S1 Text). We sought to recruit participants in the roles of organizational leadership, end-users, and project/product managers, among others. To promote participants speaking freely, we ensured that their identities would be kept anonymous throughout the course of the study and requested that managers not ask if employees participated in the study.
The semi-structured interview guide (Appendix C in S1 Text) served to explore a range of topics from Vayena and Blasimme's framework of systemic oversight as well as Gasser and colleagues' navigation aid for the ethical use of digital public health tools [20,[45][46][47].
Using this semi-structured interview guide, we conducted six pilot interviews prior to interviewing key informants to refine the content of the interview guide as well as the length and structure of the interview. We piloted with senior members of the research team and colleagues in similar roles to our key informants (e.g., clinicians, data scientists, etc.), and thus, we did not include the pilot interviews in the analysis.
The consent form was emailed to participants ahead of time, and interviews were conducted over internet-based audio/video chat on Zoom. Interviews were an hour on average and consenting participants had the option to be audio/video recorded. Interviews were then transcribed verbatim using Zoom's built-in, automated transcription feature and the adequacy of the transcriptions were assessed by VH and TS using recordings. Recordings were then deleted upon ensuring the adequacy of transcription.

Coding and data analysis
To abstract and synthesize data from key informant interviews and publicly available documents, data coding was done iteratively and through the qualitative method of abductive analysis [53]. We selected abductive analysis to move between the data and our a priori theories. The goal of the abductive analysis is to generate novel theoretical insights that reframe empirical findings in contrast to existing theories. The transcribed interviews and document analysis were described using the coding structure-specific in Appendix D in S1 Text. Additional coding domains were developed as needed during the analysis, and the key themes were identified as nodes within the qualitative software package.
As mentioned above, the primary framework used to guide analysis was Blasimme and Vayena's framework of adaptive governance through systemic oversight because it highlights a range of recommendations that promote responsible big data innovation across biomedical research, patient care, and public health. It also emphasizes the importance of directing attention to ethically controversial areas to address risks while also promoting public trust and accountability. Throughout the analysis, we also allowed thematic content to evolve independently of Blasimme and Vayena's framework.
All coding was done in duplicate by VH and TS with conflicts resolved by discussion for each document and transcript until consensus was reached. Data collection, organization, and analysis was done using the NVivo qualitative software (version 12, QSR International, Melbourne, Australia). Finally, the data from each case study was gathered and synthesized into a single brief case report to facilitate cross-case synthesis. The results of the cross-case synthesis are presented here. Recruitment and data analysis were done in an iterative process between July 2021 and February 2022.

Sample size adequacy
In keeping with case study methodology, our goal was not to speak to every single stakeholder in each case but rather to understand an issue, process, or phenomenon. We used information power as the criteria for stopping further recruitment in each case once we believed we had a clear and sufficient understanding of the phenomena in each case [54]. Due to our narrow aim (i.e., explore specific partnerships between private technology companies and public sector health organizations), specificity in experiences of our key informants (i.e., had direct involvement with partnerships during COVID-19), use of well-established theory (i.e., Blasimme and Vayena's theory of adaptive governance), and richness of interview dialogue, we found that five to ten interviews was sufficient to gain an understanding of each case in combination with the documents obtained. We then applied these requirements across our three cases.

Results
We obtained 210 publicly available documents and conducted 26 interviews on our three cases of study between June 2021 and February 2022. A breakdown on the sources and types of documents is presented in Fig 1. Most documents were news articles, marketing materials, and webpages ( Fig 1A). Most documents were obtained from company websites (Fig 1B). In what follows, we describe several major themes that emerged over our iterative, abductive analysis of these documents and interviews in relation to our guiding theories. Despite the considerable pressures of the pandemic, we found that each partnership was able to enact key aspects of adaptive governance along the pipeline of solution development to deployment (Table 1). We describe how a health emergency created pressures for our cases and how these pressures were navigated below. We first begin with the time and resource pressures created by the pandemic (theme 1), and then discuss how those pressures demand alignment early in the partnership (theme 2) and the streamlining of governance processes throughout the partnership (theme 3). We then present findings regarding a particularly important governance process, social learning, in detail (theme 4). This is followed by an outline of the specific opportunities and risks for startups during a health emergency (theme 5). Finally, we end with how teams had to navigate all these pressures and remain high-functioning (theme 6).

A health emergency ultimately creates tension with time and resources
COVID-19 has been all-consuming for public sector organizations, especially during surges when staff are pulled away from non-pandemic-related activities to maintain core functions.  -Tech company E beginning to organize around highly agile and cross-functional teams to break down silos between departments (e.g., finance, data science, product, etc.) -Tech company E had been exploring novel data streams to inform public health response for over two years prior to the pandemic, developing key internal mechanisms and safeguards -Tech company E regularly publishes in and seeks inspiration from scientific literature to guide products and consulting outputs with novel data streams, viewing peer review of methodologies to be an important gauge of quality -Public health organization F has been exploring how to modernize data science efforts and industry partnerships prior to pandemic -Public health organization F activated an emergency operations centre to centrally coordinate response related to COVID-19 -Public health organization F exploring novel methods of procurement (e.g., challenge-based procurement) -Tech company C was able to provide a high degree of customization in response to user needs (e.g., in product features, training materials, etc.) -Hospital D had to navigate potential liability with external community partners using their "instance" of the solution -Tech company E had well established measures to control for privacy (e.g., spatiotemporal aggregation), security (e.g., two-factor authentication), and quality (e.g., review of outputs at multiple stages by managers) when users were working with novel data streams to generate consulting outputs -Tech company E created automated quality assurance bots to continuously ensure integrity of back-end data pipelines -Tech company E created streamlined workflows on Slack to respond to client issues as they appeared (e.g., who discovered the problem, what causes the problem, who is working on a fix, etc.) The ever-changing nature of the pandemic also created an environment of uncertainty for the partnerships as the health organizations we studied had to respond on very short timelines to developments both in the dynamics of the pandemic itself (e.g., the emergence of new variants such as Delta) or in pandemic-related policies (e.g., changes in testing criteria). These immense demands either catalyzed partnerships with private technology companies to form de novo or for existing partnerships to expand into new areas in reaction to evolving needs, such as keeping track of which patients were COVID-positive in the hospital or who received COVID-19 vaccinations in the community.  However, there was a dichotomy to the effect of the pandemic on public-private partnerships: the same time pressures that posed challenges for the partnership also served as a catalyst -Ongoing meetings and alignment between C-Suite at hospital B and tech company A -Tech company A had standard practice of using a risk registry that would be updated through project life cycle -Patients could directly provide feedback into the virtual care platform, which was integrated into product roadmaps -Clearly measurable KPIs were developed in alignment with strategic objectives (e.g., reduce avoidable hospital transfers) -Tech company A recently completed provincial vendor verification, meeting standards for privacy, security, and interoperability -Ongoing meetings and alignment between C-Suite at hospital D and tech company C -Clearly measurable KPIs were developed where possible -Continuously evaluated expansion of features / workflows against core value proposition of solution -Tech company C continuously sought out feedback from diverse user groups to inform product roadmap and features Tech company C recently completed detailed security certification and displays key features on a web-based dashboard -Ongoing meetings between C-Suite at tech company and high-level managers at public health organization F -Tech company E continuously sought out feedback from users to inform consulting outputs and product roadmap -Tech company E recently completed security audit and will be bringing on in-house security expertise to guide work with novel data streams Acronyms: LTCH-Long-term care home; ED-Emergency department; KPI-Key performance indicators https://doi.org/10.1371/journal.pdig.0000164.t001 People will ask questions people will confirm things people will bring up ideas during the kickoff call and it just, it makes for a much better deployment I think when you have. . . really engaged project leads." (Case 2 -Interview-Customer Success) "So, and I say this because . . . a lot of times companies don't want to admit that maybe they made a mistake, or . . . they're really really busy they're overwhelmed . . . if you're not transparent from the get-go, it's really hard to rebuild the trust." (Case 3 -Interview-Customer Success) "Typically, for myself the contracts . . . is clear and explicit list of expectations so that we have something to fall back on and say, this is where we need added support . . . so that we can make sure that we are having our needs met, and providing you with enough information so that you can meet those needs." (Case 3 -Interview-Data Engineer) for them. Participants in the private sector across our three cases felt that the business value of their solutions was made much more apparent during the pandemic. For example, virtual care proliferated in March 2020 because providers needed the means to connect with patients when outpatient clinics were shut down. Moreover, the public sector organizations were able to move much faster during the pandemic to form partnerships partly because leadership had a "burning platform" to act, especially if inaction would also cause harm. Leadership buy-in was an enabler to removing "red tape" that would otherwise stall partnership formation.  "So, we would always have that continuity between that one key person who's talking both with the data provider works with the data team intensively and is involved with any meetings with the client, whenever it comes to using the data. So just making sure there's continuity so there's never that gap in communication I think it's important, and super helpful." (Case 3-Interview-Data Scientist) ". . . we did do this on a very small scale, like a pilot project before we actually did this in a larger way at one home with a very highly motivated group of individuals, and what I did was I presented at their. . . physician advisory committee meeting, and that meeting occurred once every month, and being able to rapidly obtain key information from them what worked well what did not work well almost in like a PDSA cycle format and being able to implement . . . was really the key to success here." (Case 1 -Interview-Clinical Champion)

Deployment and Evaluation
Communicate proactively and continuously, keep records and capture learnings -Transparency -Mechanisms to learn from progress -Patience, empathy, and other people skills -Social learning -Lack of alignment on core problem to be solved -Difficulty with measuring outcomes -Limited time and resources "I think like the most important thing is just ensuring that everyone is on the same page and everyone is working towards the right the right goal. If people are kind of misguided and doing things for the wrong reasons, then the partnership just falls apart." (Case 1-Interview-Clinical Champion) "We're really trying to formalize our processes and so now we meet as a team. . . bi-weekly and one of the standing agenda items is to talk about challenges, successes, key learnings. So we try to establish a culture of information sharing. . . we do it a bit more informally. . . rather than having a "playbook" that we update every time. Early and sustained alignment around the problem is core to success We found that a clear understanding of a central problem was critical to the success of our studied partnerships. Participants noted that successful partnerships were the ones where effort was made by the vendor to deeply understand the client's pain points during the sales cycle and longitudinally through the partnership.
"So each party I think really has to. . . figure out where their little Venn diagrams can overlap in the middle to the success of both. . . in a pandemic that same approach would be even more profoundly important, because speed is not on your side there. You gotta move quicker, faster, and so you better define that common space in the middle, much faster than you would in normal times." (Case 1-Interview-Incubator Lead) Once a partnership was underway, participants consistently stressed the importance of continuous transparency around motivations, deliverables, progress, and roadblocks. Transparent communications were viewed as a core enabler of trust that both parties were aligned towards solving the same problem.

". . .having that open, honest, and clear communication will allow just, you know for that trust to be built." (Case 2-Interview-Customer Success)
The success of the partnerships in our case study is also reflected in the establishment of governance structures that support early and continuous alignment through transparency and trust. Logic diagrams demonstrate the multi-level connections between partner organizations, both in healthcare (Fig 2) and in public health (Fig 3). In both cases, the client success role has high centrality within the governance structure and serves as a vital "bridge" between the partners. Beyond customer success, participants also spoke to the importance of ensuring alignment between different levels of seniority across the partnership and being able to escalate issues appropriately. One participant called this the "zipper:" "I zipper people in and I make them develop relationships. . . Those contact points determine success or failure. . .

So as an example [if] you were the director level or PM, I'd have you and your other person report monthly to [tech company A CEO] and myself. . . And you live and die together. . .[you need] the zipper, the alignment of interest, the processes to cause reporting, and the honesty to escalate when something's going wrong [to] deal with it at the right level, you can't deal with a strategic issue at an operational level." (Case 1-Interview-Chief of Staff)
The emergency context led to the triage and streamlining of governance processes Table 3 shows the wide range of tools that were used to govern the partnerships (i.e., governance processes) across our three cases of study. Participants noted that the time and resource pressures of the pandemic led to shifts in certain governance processes, such as identifying goals and allocating tasks, from formal mechanisms (e.g., project charters) to more informal ones (e.g., rapid, huddle-style meetings).

(Case 3-Interview-Chief Executive Officer)
While many governance processes were modified to be more efficient, participants also stressed certain processes that were vital and needed to be maintained. For example, stakeholder assessment could not be skipped in its entirety but leveraging the public organization's ownership and deep understanding of their problem and context allowed for one private organization to identify and prioritize "directly-impacted" groups during an accelerated stakeholder assessment. Moreover, mechanisms around what Blasimme and Vayena term

PLOS DIGITAL HEALTH
"monitoring" such as maintaining risk registries or keeping track of key performance indicators, could not be dropped altogether because they helped the partnership remain aligned and navigate emerging risks. This is especially true in the context of privacy or security issues.

"I'd say that getting approval and acknowledgment from stakeholders. . . is a step you just can't skip. . . if the stakeholder groups asked you to do something, you should have acknowledgment and awareness at a senior team level that it's being done."(Case 1-Interview-Chief Security and Privacy Officer)
An unexpected finding was that participants did not prioritize publicly sharing details of the partnership. Leaders stressed that, especially in the "information overload" context of the pandemic, publicly-directed communications about a given partnership should be meaningful and prioritized around the key needs of stakeholders.

Social learning can offset some emergency pressures
Social learning, defined as learning through observing others [55], was identified by Blasimme and Vayena as a key enabler of governance by systemic oversight and was identified across all three study cases. For the hospitals, social learning occurred both informally (e.g., Chief Information Officers would reach out to each other for advice) and formally (e.g., the local  Table 3. Common government processes utilized across our three cases of study.

Contracts
-Setting expectations between parties -Mitigates risks -Holds parties accountable for certain deliverables, on certain timelines -Examples include master service agreements, end user license agreements, etc.
"I think every decision should be as evidence as informed as possible. Usually in crisis, you, you feel that that has to go. . .so don't sign a ten year contract. . . put it on a, you know, six month contract with the option of renewal (Case 1-Interview-Chief Medical Information Officer) "We have a master service agreement with [tech company C CEO] and he agreed in our contract . . . we could extend it to anyone in our Ontario Health Team, and they named all the partners . . . But then what we did, though, was he did sign separate contracts with each of those groups that were actually coming on to our instance" (Case 2-Interview-Chief Information Officer)

Documentation
-Provides records of key decisions/events for parties to look back onto -Supports building institutional knowledge "We're using SharePoint now . . . everything is on SharePoint and we you know we work on communal documents that way as well. But definitely you know documenting methods and having them in an easy to find spot for everyone to access is, is important. . ." (Case 3-Interview-Data Scientist)

Key Performance Indicators (KPIs) and Metrics
-Varying processes done to measure progress towards a specific goal -Measuring the success or lack of towards an organization's objectives in a partnership

Meetings
-Many different styles and variations; ranges from formal presentations to stakeholder daily operational huddles -Tool to ensure transparency and alignment throughout the partnership -Provide updates between parties -Informally, the "glue" that holds partnership together Privacy Assessment -Formal process done to determine how a program/service could affect the privacy of an individual -Helps avoid the potential negative effects of privacy breach "Once we got a few hospitals, that's when you can connect the Privacy Officer at one hospital to another one and be like, oh yeah let's share notes. Until then, it was a privacy document that we created with our external privacy firm and the credibility was incomplete there . . . you can always get the hospital do all a privacy impact assessment on you, but if you make them do the entire work, you can probably assume that's a lot of resources so your sale is probably not gonna go through" (Case 1 -Interview-CEO) "From a modeling point of view. I think for the most part you always want to have, as disaggregated data as possible because it allows you to do a lot more analysis at that level. Obviously, complete disaggregation comes with huge privacy and ethical concerns so that's. . . always kind of taken off the table" (Case 3-Interview-Senior Epidemiologist)

Procurement
-Formal process of finding and agreeing to terms and acquiring services from external sources -Done through a tendering or competitive bidding process -In Ontario, broader public sector guidelines exist when buying solutions over $100,000 "The fragmented purchasing that we see in Ontario means it's very very difficult for us to grow here in Canada. . . I can only get procurement from one hospital at a time.. . .It's a very, very long procurement process. And for me to get to meaningful scale here Ontario I'm gonna have to get a lot of hospitals to buy my products, it's very hard for me to be a successful company here in Ontario . . . So, the way that we procure in Ontario certainly drive some of our emerging technology superstars in the healthcare space to go south of the border or to other countries versus to try to operate in our market." (Case 1-Interview-CEO) "I appreciate why there is a broader public sector procurement rules in place, because often, you may miss out on maybe something that actually would achieve your needs better because you're not actually considering all options, so that's why there's this forced procurement which allows for more open bidding and you may actually get better responses right. .
university's medical school co-ordinated city-wide COVID-19 response tables for hospital leadership that met as often as daily). For some participants, social learning was viewed as a bare minimum level of due diligence needed before embarking on a partnership with a vendor.
"As much as a crisis makes people make rapid decisions-don't [do that]. Take the time to at least do a, you know, an environmental scan and say like, "hey, what is everybody else doing?"" (Case 1-Interview-Chief Medical Information Officer)

Governance Processes Description Quotes
Product Roadmap -Living document that maps out the strategy, deliverables, and goals for a product overtime -Essential tool for prioritizing development of new features -Aid in achieving alignment between stakeholders in partnership "I worked with them to kind of say, "Okay, this is a product roadmap that that we think is important for you" to develop a product to meet our needs and I think they saw that that would be generally applicable to other customers in their in their market and so it was again a symbiotic relationship. . ." (Case 2 -Interview-Chief Medical Information Officer) "There's like tactical things like keeping [clients] aware of advancements to the product so like any new features that might be coming out to communicating more long-term vision . . . And also relaying that information back to us, if there is you know specific requests and so on." (Case 3 -Interview-Product Manager)

Project Charters
-Formal document that clearly outlines the project objective -Also includes a project's scope and responsibilities -The importance of such a document has been met with mixed opinions ". . .I believe it's imperative. . . it has been my experience that you have to get into some of those details and you have to write them down and you have to look at them a few times and go over them and read them out loud together in order to really make sure that you're teasing out any assumptions people may be making about what we're going to achieve together." (Case 1-Interview-Vice President) "They're almost traditional artifacts that brought peace of mind to the traditional old guard on how things are done. . . .you can get by without an official charter as long as you almost have like a, like a charter cheat sheet . . . like a one pager or some sort of living like mission statement or principles that you're guiding the implementation . . . I still think charters are important and they're ideal if you have the time to build a robust one . . . but by no means do I think that they're, they're mandatory." (Case 1-Interview-Sales)

Risk Register
-Formal document used as a risk management tool -Done to fulfil regulatory compliance for all risks identified -Includes information about each risk, the nature of the risk, and mitigation measures ". . . the kind of the risk log or risk register something that's just regularly reviewed and updated based on how the project is going so you can get ahead of things or mitigate the things as best as you can." (Case 1-Interview-Sales)

Security Assessment
-Formal process to test an organization's security preparedness -Checks for vulnerabilities within technology systems and business processes -Can include recommendations to lower the risk of future attacks "I'm talking about cybersecurity. Typically when you do these things you like to do threat risk assessments you like to do penetration tests and things like that. I'm given limited time. You may not always have the luxury of doing the things that you would like to do to ensure you know that level of due diligence. . . so what controls are in place to ensure that, for example, anyone who needs access to the system is access is granted on, you know, least, least privileged required to do their job. . . . ensuring that security has been implemented to align with, you know, hospital policy as an example. . . things like you know password policies is a simple example." (Case 1-Interview-Vice President)

Stakeholder Engagement
-A process done to identify the needs of key stakeholders -Includes the identification, analysis, planning, and implementation of actions in which key groups are present ". . .community health groups or community groups, organizations they already exist. . . you don't have to make up a new body. They already exist. They are already doing work. Go talk to them and it could be as simple as just having a phone call with them. You don't need to set up like a complicated consultation body and process. . .you do need to make sure that those voices are embedded throughout which obviously demands some kind of governance. . . rethinking of those mechanisms, but yeah, they already exist-so use them." (Case 3 -Interview-Policy Analyst)

Standards and Certifications
-Certifying products to meet standards (e.g., ISO) -Helps signal buyers and competitors the adherence to best practice -Determined by independent third-party "[Tech Company A software] is now listed as validated . . . for virtual care. . . and meets all of the provincial privacy, security, interoperability, and technical requirements. The public health organization we studied also had a small team dedicated to learning from jurisdictions worldwide, but it was less clear how to take these learnings and apply them to their own context compared to the hospitals. "I think there was an effort to constantly learn from other countries. . . there's a group whose only task is to look at other countries, what they're doing, and how they're performing as a result of what they're doing. . . and that project has been ongoing for at least probably since the summer of 2020." (Case 3-Interview-Senior Epidemiologist) For the private technology companies, social learning took the form of press releases, when deemed appropriate, to signal key milestones (e.g., launching new products or services). Conversely, it also involved scanning for other offerings in the market to understand and crystallize their unique value proposition as a company. For one partnership, they noted that social learning was formalized by the nature of their membership in an innovation accelerator, where they delivered and heard presentations about successful and unsuccessful healthtech partnerships from other participants in the program.

Startups have a vital role to play in emergency response, but growth can introduce new challenges
Since the companies studied were startups, we found that they were highly attuned to their local contexts and could often be highly flexible and adaptable to create bespoke solutions for their partners. Several participants from the hospitals contrasted their previous experiences with major electronic medical record (EMR) companies based in the United States, positively reflecting on how their perspectives as end-users were much more valued and regularly informed product updates compared to the EMR companies.
"I think with the big vendors, the other thing is making sure that I'm not getting the short end of the stick where I'm paying a lot but actually not getting service. . . whereas I think those discussions are easier with a smaller company right because they just have more flexibility" (Case 2-Interview-Chief Information Officer) All three startups studied scaled dramatically (i.e., doubled, tripled) during the pandemic due to the increased demand for their solutions. While on the one hand, the increased demand and revenue is positive and reinforce their value, aspects of pandemic-fueled "hypergrowth" were also challenging. For example, one dimension of growth was around staffing and building capacity to complete core tasks in response to increased demand, such as responding to customer support tickets or onboarding new users. Given the urgent, and oftentimes bespoke, nature of client demands during the pandemic, the companies we studied also had to navigate how to maintain their core value proposition.
"Each client wants a slightly different version of what you're trying to offer. And so you start to get this just like infinite list of things that you can build against but it only applies to one client at a time, and building like bespoke things at scale is just not possible." (Case 3-Interview -Product Manager)

There's no partnership without healthy, motivated teams
The intensity of the pandemic timelines and workloads universally created burnout and a high degree of turnover among teams in both the public and private sectors. Effective interpersonal skills and emotional intelligence, especially empathy, was viewed as an enabler of effective partnerships.
"I'll typically relate back to you know I get it, I'm a healthcare professional myself, you know, things come up, emergencies come up. . . I think that that perspective allows me to be there in a more supportive way because I do understand that their environment can be very hectic and can be ever-changing." (Case 2-Interview-Customer Support) Participants also spoke to the need for visibility into governance processes. Leaders felt the onus to be active throughout the partnership and diagnose areas of potential misalignment quickly. Having key governance processes, such as discussions on ethical issues or company direction, being highly visible and inclusive helped to keep teams motivated and engaged, especially in virtual and thus potentially isolating or siloed work environments.
"It really speaks to you know being in that supportive environment where we can listen to everyone's concerns. . .So the fact that we are able to bake in, you know, being considerate of a client's time and everyone's time but still take the due diligence in order to be able to say can we [navigate this challenge] differently and more responsibly, I'm very happy to be a part of something like that.

(Case 3-Interview-Data Engineering)
Ultimately, one of the best remedies against burnout and promoters of effective team functioning was that participants on both the public and private side of the partnerships we studied found that their work during the pandemic was valuable and impactful.

Discussion
We found that key aspects of adaptive governance were adopted in three distinct partnerships between health organizations and private technology companies to deploy digital technologies as part of COVID-19 pandemic response. While the emergency context created considerable pressures with respect to time and resources, the partnerships we studied were able to use a wide range of governance structures and processes to ensure they were continuously aligned on shared goals. We found that governance processes were continuously refined and streamlined in response to logistical pressures but that certain processes, such as stakeholder engagement and monitoring for the emergence of new risks, were identified as essential. Most of the insights were drawn from key informant interviews, as publicly available documents did not frequently address the inner workings of these partnerships. Taken together, our findings suggest that effective and responsible cross-sector partnerships to deploy digital technologies during health emergencies are possible with several elements that enabled successful collaboration outlined in our findings. However, some learnings can be used to design and improve future health technology (healthtech) partnerships.
Both Blasimme and Vayena's framework of adaptive governance through systematic oversight and Gasser et al.'s navigation aid were helpful when applied to our real-world cases [20,[45][46][47]. Although Blasimme and Vayena's framework was designed specifically around big data research, we found that many concepts were applicable to digital technologies more broadly. For example, Blasimme and Vayena defined "reflexivity" as an awareness of the harmful, classificatory impact of machine learning algorithms, but we found reflexivity to be applicable in broader terms such as an awareness of how deploying virtual care technology could cause harm due to the digital divide [56][57][58]. Our key informants were able to enact many aspects of these two frameworks into practice, even though they were not explicitly used or perhaps were not aware of them. Thus, an important area for knowledge translation could be to improve the visibility of these frameworks to key actors engaging in partnerships via integration in formal education or upskilling. Participants often spoke to best practices from their respective fields as guiding their work, such as plan-do-study-act (PDSA) cycles for clinicians and the Information Technology Infrastructure Library and The Open Group Dependability Through Assuredness TM (O-DA) Architecture Framework for informaticians [59][60][61].
While frameworks and best practices can be helpful while governing private sector partnerships, our findings also emphasize that firms should not necessarily chase a "silver bullet" governance arrangement. Rather these higher-level guidelines on governance structures and mechanisms should be interpreted as such. Across our cases of study, we found a high degree of "bricolage", or improvisation, by actors in both the private and public sector to navigate their partnership [62,63]. Given the importance of experimentation, it stands to reason that partnerships that are in place prior to an emergency can leverage the "peacetime" environment to conduct the necessary experimentation to determine what works best in their own context. Experimentation around governance models has long been explored in the realm of climate governance, but is relatively new in digital health [64].
Our findings also demonstrate the importance of viewing digital technology deployments in health as complex, socio-technical systems. It would be incorrect to assume that private sector actors can "parachute in" and fix problems, especially in a crisis. As digital technologies change workflows, teamwork across a wide range of stakeholders is needed to repair these workflows and move forward to a new equilibrium [65]. The success of the partnerships in our study hinged on meaningful engagement across the entire pipeline from problem identification to solution deployment and evaluation. Health organizations should focus on their understanding, and ultimately ownership, of the core problem to ensure alignment for the partnership. Marwaha and colleagues highlight the role of key players within the health system such as clinical champions and executive sponsors as drivers of this alignment, but our work also demonstrates that it becomes important who they are connected to on their partner side and how [66]. At a broader level, human resource and capital considerations (e.g., burnout, people skills) were not captured in our guiding frameworks but should not be ignored as a key driver of effective and responsible governance [67].
The healthtech landscape in industry is largely bimodal, with major, incumbent multinationals (e.g., Google, Apple, Microsoft) on one side and startups on the other [41]. A key strength of our work was to focus on the relatively understudied role of smaller companies and startups in the context of a public health emergency. Our findings support existing literature that startups are highly agile and able to tailor their offerings to local needs, which makes them valuable contributors to crisis response [35]. However, the pandemic has created an additional layer of stress on firms that already have to navigate uncertain business conditions and reveals areas of improvement to better support these vulnerable entities. Healthtech startups must be careful to maintain their core value proposition and grow sustainably, even when an emergency context provides an impetus for new partnerships to form with hospitals and public health organizations. In many provinces in Canada, hospitals and public health organizations are public entities that are subject to public sector accountability acts. These acts outline rules for procuring technologies that are meant to promote fairness, competition, and the responsible spending of taxpayer dollars. However, many respondents felt that these guidelines make the procurement process too slow, cumbersome, and stifle the ability of non-incumbents to compete given lengthy sales cycles-even outside of an emergency context. To address these issues, the United Kingdom created a Procurement Policy Note in March 2020 that outlined criteria for public sector purchasers to issue direct awards or launch competitions with accelerated timescales [68]. Outside of emergencies, innovation accelerators embedded in health organizations can offer a testbed for governance experimentation, increase risk tolerance, and formalize social learning. The National Health Service (NHS) has developed a series of pathways to foster the invention, development, and implementation of innovations, including digital technologies [69,70]. These pathways exist to, among other functions, accelerate otherwise labyrinthine procurement processes for small and medium-sized companies, create pilot sites for deployments in partnership with academic medical centers, and support a class of fellows in scaling innovations across the country [69,70]. Similar accelerators exist at academic medical centers in the United States, such as at the University of California San Francisco and Mass General Brigham [71,72].
The interpretation of our findings should be considered in light of certain limitations. Firstly, the three cases of study in this work do not exhaustively cover every application of pandemic technology but are broadly representative of major categories (i.e., clinical, operational, and epidemiological). Secondly, the partnerships studied here can be deemed successful (i.e., they all reached the deployment phase) and maintain ongoing business relationships. Thus it is possible there is an element of social desirability as well as recall bias among our key informants leading to an overly positive appraisal of the partnership and events therein. Throughout our interviews, we attempted to address this issue and obtain critical perspectives by probing participants on what they thought could have gone differently or be improved as well as reassuring them that we would protect their anonymity (i.e., both personally and organizationally) to help them speak freely. Most interviews were conducted with managers and other participants at senior levels within the organizations of study, but we believe these perspectives are appropriate since it is leadership who generally steer the direction and priorities of partnerships. Finally, our study relied on publicly available documents-which we found were helpful for outlining that a partnership formed or key outcomes but not often why certain decisions were made and how those outcomes were achieved. This is in contrast to a popular study by Powles and Hodgson who obtained documents (e.g., contracts) on the partnership between the NHS Royal Free Trust and Deepmind through a freedom of information request [22]. We addressed this limitation in publicly available documents by conducting extensive interviews with key informants. These interviews were contingent on goodwill and building trust with our key informants by conveying that our study sought to explore how to ultimately improve the functioning of future partnerships. A freedom of information request can quickly undermine this goodwill. We caution that researchers consider the benefits and drawbacks of such an approach.
The limitation on publicly available documents also speaks to an important area of future work. When asked about transparency, our key informants stressed that publicly-directed communications about a given partnership might not be appropriate because communication should be meaningful and prioritized around key stakeholder needs. This finding creates tension with the prevailing view in the literature that transparency is essential for maintaining social license [11,73,74]. Indeed, there has been criticism of public registries of artificial intelligence algorithms in Amsterdam and Helsinki since they are not, in isolation, a sufficient governance solution to ensure ethical use [75]. Unfortunately, we were unable to provide guidance around how those communications, and to a further extent public engagement, should be operationalized. Standards, regulations, and expectations around transparency are also highly likely to be variable between jurisdictions. Future work should explore how to effectively and pragmatically foster transparency and public engagement around partnerships between health organizations and technology companies, especially in the context of a health emergency. Finally, we believe that as partnerships between digital health organizations and technology companies increase in frequency, future work should continue to take conceptual frameworks and seek to explore their relevance and impact empirically in real-world contexts.

Conclusion
Strong public-private partnerships are essential to maintain resilient health systems. Despite the considerable time and resource pressures of the COVID-19 pandemic, we found that key aspects of adaptive governance, such as social learning and flexibility around governance processes, helped foster effective and responsible partnerships between health organizations and technology companies. COVID-19 has accelerated the digital transformation of health systems globally, and a return to the pre-pandemic normal is increasingly unlikely. Stakeholders must leverage the learnings from the ongoing pandemic to navigate not only future crises but also as private technology companies continue to expand their role in health.