Top-level questions for the research
How effective is the App against COVID-19? Can it be quantified?
Does the App, based the its current implementation status and usage that we find, have unique opportunities to enlarge its impact on the pandemic or further reduce downsides or risks? What recommendations can we make to the App Creator in the context of current world-wide best practices?
Target audiences and their needs
Our research is targeted towards the following audiences:
The general public wants to know which App they should use, why they should use it, and that it is safe for them to do so. Should there be risks or downsides: what are they?
Public health authorities want to know the best practices for Apps helping with the pandemic and understand how they have been implemented around the world, with the goal to adapt them to local circumstances.
App developers and operators may be unfamiliar with some approaches (technologies, operations or governance) that have been successful for other App deployments around the world, and would like to learn how to adapt them for their App.
Information sources for an App
Our research into an App is based on the following information sources, to the extent they are available:
Information from third-party audits, in which a qualified independent third-party entity has the full cooperation of the App Creators, and full access to all relevant people and information, in order to assess all aspects of the App.
We consider third-party audits the best source for reliable information about an App. Relevant parties include experts in epidemiology, public health, technology, operations, governance and civil rights.
Completeness and correctness
Analyzing an App is a complex undertaking. As for any complex undertaking, unfortunately mistakes not only can happen, but are virtually certain to happen. So regardless how diligent a job we try to do, we must work under the assumption that our analysis will be incomplete and incorrect in part.
When this happens, we apologize, and would like to invite you to tell us by filing an issue in our issue tracker. We will review the issue, and depending on whether we concur with your assessment or not, make suitable corrections to this site.
Even after we close the issue in the issue tracker, it and any resulting discussion will remain available there for review by you and others.
Note the site’s general disclaimer.
We use this Questionnaire for an App.
Technical assay of an App
We perform a technical assay if enough of the source code of the App is available, and the cost/benefit ratio appears beneficial related to our available resources.
The technical assay is directed by the questionnaire that we attempt to answer.
The details of the technical assay highly depend on the features, architecture, and availability of information about the App.
Summary of the assay for an App
- Impact on the pandemic.
- Avoidance of downsides or risks.
These are separately rated.
For both ratings, the same symbols are used.
For example, a hypothetical App may have the following summary:
Impact on the pandemic
which should be interpreted as:
- In the view of the App Creators, the App has a significantly positive impact on fighting back the pandemic (green arc upper-left),
- while a 3rd-party assessment disagrees (red arc lower-left), with an explanation that is given in text to the right.
Avoidance of downsides or risks
which should be interpreted as:
- An audit on potential risks and harms was conducted, which found a substandard result (full circle in yellow), with an explanation that is given in text to the right.
Sources for the methodology
Chaos Computer Club: 10 requirements for the evaluation of “Contact Tracing” apps. 2020-04-06.
Greenwood, Dazza; Gregory Nadeau; Pagona Tsormpatzoudi; Bryan Wilson; Jeffrey Saviano; and Alex “Sandy” Pentland: COVID-19 Contact Tracing Privacy Principles. MIT Computational Law Report. April 5, 2020 and updated April 30, 2020.
Laux, Christian: Checklist for Contact Tracing Apps. Published to MyData Global Slack. 2020-03-26.
Maddocks for the Australian Department of Health: The Covidsafe Application. Privacy Impact Assessment. April 24, 2020.
Vokinger Kerstin N.a; Nittas Vasileiosb; Witt Claudia M.c; Fabrikant Sara Irinad; von Wyl Viktorbe: Digital health and the COVID-19 epidemic: an assessment framework for apps from an epidemiological and legal perspective. Swiss Medical Weekly, 2020-05-17