Hub technology

The Hub is cloud-based software that provides access to health-related algorithms and models. It offers clinicians, consumers, hospitals and government decision-makers free use of a library of models, with the ability to process data inputs and generate outputs. Models are accessible via a website for self-service, a REST API to allow technology integration, and batch processing that allows authorised users to send multiple sets of inputs as a single batch for efficient processing.

For data scientists, researchers and others contributing algorithms and models, the Hub aims to support deployment of their work without the burden of technical support and system software maintenance falling on the creators. The hope is that the stable, publicly accessible, automated and scalable platform encourages broader use of the models, and also frees up algorithm developers to make improvements or focus on new challenges.

Process for algorithm assessment

In the current process, model developers can submit an algorithm for evaluation and hosting on the Hub. Algorithms are subsequently reviewed by a multidisciplinary Governance Group with a wide range of experience, before they are included on the Hub.

Initial identification of candidate models involved direct engagement with the health sector, proactive enquiries from model developers, and ongoing scanning of the international literature and evidence. The Hub development team initially used focus groups and webinars to engage with potential model end users and contributors. Models that align well with the purpose of the Hub and can provide evidence of quality and adoption are progressed through to the review process, prioritised by an assessment of speed to value.

Review processes involve the following:

• The use of model cards to summarise the opportunity and inform an initial conversation with the Governance Group (Mitchell et al. 2019).

• Long-form model documentation through an ‘Algorithm Information Request’, as below, for each model, with appropriate supplemental material. The current version is provided on the Hub website (https://bit.ly/2ZUg53a).

• Revisions are made to the documentation as required, often in collaboration between the submitting participants and Governance Group members.

• For endorsed models, a summary of the governance documentation and discussion tailored for algorithm users is prepared and published with the algorithm on the Hub website, and elsewhere as appropriate.

The Algorithm Information Request was co-developed with the Governance Group. We started by preparing a long list of questions that were relevant for the assessment of models and algorithms for COVID-19, drawing on a range of sources (Moons et al. 2015, 2019; Collins et al. 2021), and refined these as the first algorithms were scrutinised. Informed by the experience and expertise of the Governance Group, and the various guidelines discussed below, we organised the questions raised around some specific areas of interest.

On account of Aotearoa NZ’s successful COVID-19 elimination strategy in 2020, the experience of the pandemic at that time differed to many other countries. In particular, very low case numbers meant there was insufficient data to validate most COVID-19-specific models and algorithms for the Aotearoa NZ population. Decisions about the scope of the models and algorithms to be deployed on the Hub were made with an eye on Aotearoa NZ’s unique situation and needs.

Models included on the Hub

Over 30 models and calculators were reviewed for inclusion on the Hub. This included a range of standard medical calculators, which didn’t require extensive discussion. Of more interest to the Governance Group were models related to the pandemic spread and relative risk calculators.

The models operated at different levels of granularity and consequently the issues raised in their consideration were diverse. One of the first models considered and included was the COVID-19 stochastic branching process model (Hendy et al. 2021) developed by researchers from Te Pūnaha Matatini to simulate the spread and effects of COVID-19 in Aotearoa NZ and support a range of planning and policy requirements. Used extensively for scenario planning, this methodology is appropriate while case numbers are low and in the absence of widespread community transmission, as was the case at the time in Aotearoa NZ. Another was nzRISK (Campbell et al. 2019), a preoperative risk prediction tool that estimates the risk of mortality post-surgery; developed and validated in Aotearoa NZ for adult patients undergoing non-cardiac surgery, nzRISK aids in processing the anticipated backlog of unmet surgical demand. The Governance Group also evaluated and included a surgical scheduling algorithm (Soh et al. 2020) which performs a smart optimisation to book a surgical waitlist against available theatres and session slots. A further example was the Measuring Multimorbidity Index (Stanley and Sarfati 2017), which has been developed in Aotearoa NZ as an index for short-term mortality risk using chronic conditions identified from routine hospital admission data. While other comorbidity indices exist, they are largely developed from international data and so may not reflect Aotearoa NZ’s unique population and healthcare settings.

shows an overview of the categories of algorithm included as well as an example of each. A more detailed description of the models included, and associated governance decisions are provided in supplemental reading.

Ethical considerations

The Governance Group focused on identifying ethical risks and suggesting mitigations where possible. We did not apply a rigid ‘ethics checklist’, nonetheless some familiar ethical concerns and associated principles emerged.

Following familiar consequentialist principles, we aimed to identify the burden and benefits posed by an algorithm and its proposed implementation. Attention to implementation decisions are crucial. Consider nzRisk, a simple and transparent tool to inform patients’ choices (Campbell et al. 2019). The tool’s scores reflect existing background inequities in Aotearoa NZ’s health system, accurately predicting poorer surgical outcomes for Māori. This does not represent a flaw in the algorithm as the disparities are real, lived, and so embodied in the data. However, were the tool used to prioritise patients most likely to have good outcomes, it would exacerbate those inequities. Implemented in some ways the algorithm is valuable for helping patients and their family/whānau make informed choices; used another way it might worsen existing inequities.

Algorithms should offer genuine benefits and those benefits must outweigh any risks they pose, and these potential risks are broad. The Hub’s governance approach ensured that through open discussion and consideration the risks posed by the use of an algorithm were clearly identified.

Many ethical concerns about algorithms flow from the value of respect for persons, from, that is to say, broadly deontological principles. Standard requirements for transparency and explainability, for instance, rest on the idea that those affected by algorithms must be able to make informed choices. We know too that many patients want health care decisions to be made by humans (Longoni et al. 2019). That preference may rest on the recognition that empathy is important in health care and algorithms cannot empathise and that no algorithm will perfectly capture the richness of any individual’s life, or their clinical, social, biological or demographic characteristics. While some patients may at least occasionally prefer models which have better than clinician accuracy (Pezzo and Beckstead 2020) overall many health consumers are uneasy about processes which do not leave a role for human interpretation. We must respect those genuine concerns and the inherent limitations of algorithms, recognised in Aotearoa NZ’s Algorithm Charter (Stats NZ 2020).

Other ethical concerns overlap with other areas of review by the Governance Group. The Hub’s ethical review canvassed concerns about broad issues of justice and fairness, for instance motivated by concern that algorithms might cause or exacerbate unequal access to health resources. These are outlined elsewhere in this paper.

Many of these issues require that different concerns be balanced. That is most obvious in the case of burdens and benefits, but even some conduct which appears to violate apparent rights against discrimination may be permissible if, in the words of section‘Conclusions and Recommendations’ the Bill of Rights Act, they can be ‘demonstrably justified in a free and democratic society’. To have a right, on one view, is to have a claim upon which one can insist. At the ethical coalface, matters are rarely so simple.

Overall, our approach can be seen to have followed familiar paths–autonomy, beneficence and nonmaleficence, and justice–promoting both comprehensive and consistent review across projects.

The context for Aotearoa New Zealand: Māori perspectives

In the context of providing an algorithm platform for Aotearoa NZ, specific consideration was given to the perspectives of Māori as tangata whenua. The approach is relevant to other indigenous peoples too and should be examined routinely.

The notion of indigenising algorithms involves multiple dimensions. The design of AI and machine learning (ML) systems should feature a high degree of control by Māori over what the system will look like, such that their development and implementation is relevant to Māori and serves Māori aspirations. The systems must perform well for Māori, ensuring that there is at least an equivalent capacity to benefit them. They must also satisfy Māori data sovereignty principles (Te Mana Raraunga 2018) while comporting with or expressing Māori ways of framing situations and acting in the world (Hudson et al. 2017). However, the converse situation does not honour Te Tiriti obligations. The impacts of these latter kinds of systems are often poor and can be harmful for Māori.

Acknowledging the distinction identified in Section ‘Ethical considerations’ about intended use, there was significant concern about the possibility of exacerbating existing health inequalities by the use of models that are often trained on data from the same systems where these inequities exist. There were sometimes untested assumptions regarding the suitability or impact of models for use by Māori and Māori organisations. Sometimes this came from problematic suggestions for use cases. And sometimes Māori populations might not have been considered as separate groups in a population (or these were forthcoming features planned for newer versions of the model).

While the space for indigenising algorithms was relatively limited by the nature of this project, this work provided an opportunity to discuss issues relating to safe algorithm use from a Te Tiriti perspective, and a separate equity lens, and there were attempts to orient developers and users to these obligations and risks. The Algorithm Information Request form filled in by those submitting their models for inclusion on the Hub required the researchers to respond to core questions:

• What are the relevant Māori considerations for the development and use of this model, and how have/are these being addressed? Consider specifically how this work can uphold the principles of Te Tiriti o Waitangi with reference to participation, protection and partnership.

• Describe how appropriate decision making and community engagement has taken place.

• Does the model explore, or is it able to detect, differences in outcome by population subgroup e.g. ethnicity, gender, age? Please include a specific Māori lens in your response.

• Has the algorithm been tested for differential accuracy or validity by population subgroups e.g. ethnicity? (Comment with respect to factors such as goodness of fit, performance metrics, treatment of missing data.) Please include a specific Māori lens in your response.

• Is there a potential for disproportionate benefit or disproportionate harm to one group or another in applying or interpreting the results? How do you propose to mitigate this? Please include a specific Māori lens in your response.

Although responses were inconsistent in quality, their completion (along with the face-to-face interview researchers had with the Governance Group) required explicit consideration around how their models may have the potential to mitigate or perpetuate existing disparities in healthcare outcomes or quality of care. In particular, researchers were asked to consider their methods for dealing with missing or unique data, potential for differential performance by population, presence of pre-existing bias, and suggest remedies to uses that might give rise to inequities. The Governance Group also made a decision that no personal information be collected in the use of any of the models. Finally, while the models had already been created, we used the ‘Overview’ page for a model to highlight possible unintended inequitable impacts that might be caused by certain use cases of that model.

Legal and privacy perspectives

Key legal considerations for the Governance Group centred around privacy and intellectual property rights. Human rights law considerations under the Human Rights Act 1993 were addressed when considering the ethical implications of an algorithm, as discussed above.

Privacy law concerns for health-related algorithms can arise in relation to the collection and accuracy of data used for training algorithms, as well as with how personal information is used within an algorithm. The Algorithm Information Request form included questions about how personal information may have been used in relation to model training as we as potential use. While many Hub applicants stated that their models did not involve the use of personal information such assertions were typically probed by the Governance Group, recognising that most health records are built upon personal information. However, it was not always possible to get assurance as to how personal information in training data had been collected and used, or whether it had been appropriately interrogated for accuracy and bias issues.

Ideally, those submitting information about their models would provide full details and evidence of their training data, the extent to which personal information was included, how that information was obtained and how issues like potential historical bias in the data had been addressed. Where appropriate information is not provided or there are concerns about privacy-related matters then those models should not be released.

Privacy concerns for the Hub itself were mitigated by restricting users from entering personal information when using models and by ensuring any outcomes inferred by the software following a calculation were not stored (Terms of Use and Privacy Policy). This allowed for the expediency required in a pandemic, but future enhancements of the Hub and similar initiatives will need to consider this issue closely.

Details of intellectual property right ownership and applicable licence terms were also requested in the Algorithm Information Request form. Such information was not always forthcoming. In other instances, commercial confidentiality concerns meant algorithms were not able to be made available on the Hub.

The Governance Group was not required to advise on broader legal risks associated with the Hub because such matters were dealt with by Orion Health Holdings Limited’s in-house legal team. As a result, it was sometimes unclear whose legal interests the Governance Group was supposed to consider. Liability issues were addressed in the Hub Terms of Use, which include disclaimers that the models are not intended to be used as a substitute for medical advice and that their predictive outcomes may not be accurate. Similar governance groups in future might like to more clearly define from the outset whose legal interests should be considered, so that relevant risks can be more readily identified and addressed.

Operational perspectives on the role of algorithms

Whilst clinicians may not always have to validate outputs of intelligent systems (AI Forum NZ 2020) it remains instructive to be vigilant about issues such as patient privacy and agency in addition to consent and transparency, particularly in relation to building trust and safeguarding public good interests (Powles and Hodson 2017). Transparency is at the heart of digital decision support (Larsson 2018) and highlights the need for clear explanations, in language patients and their whānau can understand, that can be integrated into shared decision-making (Magalhaes and Hounslow 2019; Heyen and Salloch 2021; Stahl et al. 2021). However, currently health consumers are predominantly passive agents (Stahl et al. 2021) who are rarely invited to share their insights and lived experiences in co-design digital partnerships (Digital Council for Aotearoa New Zealand 2020). When considering possible models for inclusion on the Hub, the Governance Group framed conversations to explore the developers’ awareness of the possible impacts of their models on different patient and family/ whānau cohorts across different contexts. Te Tiriti informed conversations about Māori patients and their whānau were informed by Te Kotahi’s Mana-mahi governance framework as it has been applied to AI (West et al. 2020) and presented in the preceding Māori perspectives’ section.

Understanding of the more general operational considerations for algorithm selection evolved over time, as the platform was broadened from its initial COVID-19 based frame. Broadly, algorithm selection required consideration of internal and external validity, as well as the potential real-world circumstances of usage. Internal validity was the most straightforward to address, in that most algorithms selected had undergone a scientific peer review process. This was reinforced by the ability to inspect the algorithms and interview the researchers directly. Whilst publication does not always guarantee appropriate methodology has been used or is reported (Dhiman et al. 2021) this aspect should continue to improve with the recent introduction of reporting standards for regression-based algorithms (Moons et al. 2015, 2019) and the proposal to extend this to AI models (Collins et al. 2021). Further thought needs to be given to whether relying on pre-existing peer review and publication standards may present a barrier to presentation of models from non-academic sources.

External validity proved more difficult to examine. This was particularly the case with COVID-19 risk prediction models that were derived from overseas datasets, with populations and healthcare systems that differ from Aotearoa NZ. Questions of generalisability and calibration of these models (Calster et al. 2019) both with respect to geographic differences and temporal evolution of the pandemic, were difficult to resolve. An example of this is the evolution of the pandemic both in terms of new variants of the virus and the effect of vaccination on disease transmission and severity, neither of which were considerations for models derived in the earliest phases of the pandemic.

In the end a pragmatic approach was taken, in the knowledge that post-derivation calibration of these models might not be feasible for the Aotearoa NZ population until case numbers were much higher. An additional focus was exploration of the inclusion or treatment of predictor variables representing ethnicity and social determinants of disease. Derivation or training of models from datasets that exclude marginalised or socio-economically disadvantaged communities, or discriminatory treatment of such variables in algorithm production, can result in encoded bias. This was of particular concern for those algorithms that might lead to increased inequity.