Measuring the impact of COVID Alert with metrics
Change is the only constant, especially when it comes to a pandemic. Like most are experiencing, I’ve had to get used to a bit of a different lifestyle – going from listening to obscure bands in eclectic spots across the GTA, to settling down with a Margaret Atwood novel. My Saturdays look very different.
Change is what we can rely on right now, and we’ve truly taken this to heart when it comes to developing the COVID Alert app. We continue to rescope, reshape and reimagine what the app might be as we learn more about the virus and how people’s day-to-day lives are changing in relation to it.
As with any change that goes into the service, we make sure it aligns with the two main principles that have been guiding us all these months:
- The service must work well and
- The more people that use the service, the more impactful it will be.
Understanding if the service works well
The service has been out in the world for about six months. We’re now at a point where, to make bigger decisions about it, we have to dive deeper into that first principle.
To understand if the service works well, we need to learn if the app is functioning as intended on a day-to-day basis, and if the overall service is having an impact on the pandemic. We can’t fully do that without data. This is why we’re introducing metrics.
As always, it’s a balancing act. To make sure we get the app into as many hands as possible – addressing the second principle – we’d need to collect the metrics while continuing to protect people’s privacy and maintaining the trust we’ve already established.
Turns out, with some help from colleagues, including those at the Office of the Privacy Commissioner of Canada and the COVID Alert Advisory Council, we can do it without compromise.
Does the collection of metrics impact people’s privacy?
We regularly brief the COVID Alert Advisory Council to get advice on promoting a safe and effective app. The Council is made up of 11 members with various expertise and perspectives including health, privacy, data governance, science, and innovation. Among those are Jean-François Gagné and Carole Piovesan – contributors to this blog post.
They helped us make sure we were approaching the collection of metrics in the interest of privacy. So that’s why we’ve taken an aggregate approach to collecting this data.
What does that mean? Well, to collect metrics in aggregate, we’re using a tally method. Each time one of the tasks we’re measuring is completed, it gets registered as an event. (For example, selecting a province or territory is an event, and so is receiving an exposure notification.) Anytime one of these “events” happens in the app, the app lets the server know, and the server counts it as a tally mark. Once a day, each of the events are tallied and we get the total number for that day. No other information is available from this method. The server does not collect or retain specific information about the device. We have no way of knowing who it came from – or anything else for that matter.
What exactly is the app collecting metrics on?
To better understand if the app is working properly on a day-to-day basis, we’ll be collecting metrics to tell us how many devices are performing the tasks we expect a working app to perform. This includes how many devices performed a background check each day, or how many devices successfully completed the onboarding process.
This information will give us a deeper understanding of how we can build the app to better suit the needs of those who are using it. It will also help us spot irregularities and respond to bug reports and issues more efficiently.
Zooming out a bit, we also want to understand COVID Alert’s impact on slowing the spread of COVID-19 in Canada. For example, we’ll now be collecting the number of devices that receive an exposure notification, and then enter a one-time key. This would indicate that the service is effectively alerting people who are COVID-19 positive to take action.
Understanding this will help us answer larger questions around how we might improve the overall service, including how people receive one-time keys. It also gives us better insight into the app’s impact on the pandemic and what this might mean for reopening the economy.
You can see a full list of the specific metrics the app is collecting in the COVID Alert Privacy Notice.
What does this mean for people who use the app?
Though we’re adding changes to help us make the app and the overall service better, people who use the app won’t see any difference in how they interact with it.
Builders and users of the app
At our day job, we’re a team of dedicated developers, designers, policy hackers, communications experts, product managers and many others. But beyond that, we’re also users of the app, who want to see life get back to normal – obscure live music concerts and all. To get there, we want to make sure our loved ones and communities have the best tools possible to stay safe, in a way that protects their privacy.
The introduction of metrics into the app does not change that. We’ll continue to advocate for all users of the app, and protect the privacy and security of the people who choose to use COVID Alert as another way to stay safe during the pandemic.