Principles

Here are some things that CDS believes about evaluation:

  • Every delivery is subject to evaluation. Evaluation is a constant in the product delivery lifecycle; it takes place in every story, sprint and every phase.
  • We evaluate to make decisions, understand our progress, and demonstrate our return on investment.
  • Evaluation covers delivery, product performance, partner capacity and outcomes.
  • Every delivery team does evaluation. Not all products are alike; delivery teams decide what’s best based on needs, service, phase, partner, solution, time. We make hypotheses about evaluation, and iterate as we learn.
  • There are people in teams who lead on evaluation but it’s a multidisciplinary effort.
  • Evaluation is a collaborative activity with partners. It’s one of the things we hand over to partners for continuous improvement.
  • Plans and results for each product and team are available for CDS and the partners to view, and are linked to from a central place.

Categories of evaluation metrics

CDS collects qualitative and quantitative metrics in 4 categories:

  1. Delivery
    The wellbeing, functioning and productivity of a team
  2. Partner capacity
    Partners ability to work with new principles, technologies and techniques
  3. Product performance
    The performance and efficacy of the product and its features
  4. Outcomes
    The impact of the product or service on people’s lives

Evaluation in the delivery phases

Evaluation metrics and reporting increases in precision and frequency at each phase:

  1. Discover
    Exploring what the metrics should be, how to get them based on research, and existing baseline information about user needs
  2. Alpha
    Testing and some tracking of the metrics through prototypes
  3. Beta
    Benchmarking and reporting the metrics by releasing minimum viable thing
  4. Live
    Reporting the metrics and making continuous improvements to the live product

Examples of evaluation metrics

These are some examples of measurements that could be collected during each phase for each metric category:

Discovery
Delivery -
Information to understand how well we are using our resources in Discovery.
  • Cost of delivery team
  • Resources dedicated to recruiting users
  • Cost avoidance due to building or buying
Capacity -
Baselining information on how a partner works pre-CDS.
  • Frequency of code release in partner organization
  • Lead time for changes (time from commit to production)
  • Time to restore service (e.g., from an outage)
  • Code change failure rate (% changes causing failure)
  • % of positions on CDS team matched by member of partner team
  • Key blockers to design research, iterative development, or open methods
Performance -
Baselining existing information on existing product or service performance.
  • Drop-off rates of key tasks required to complete the service
  • Time to complete product or service interaction
  • Content reading level
Outcome -
Information about what the public benefits might be.
  • Only 20% of eligible people access benefits
  • It takes 120 days to process each application
  • 14,682 appointments are missed each year
Alpha
Delivery
Information to understand how well we are using our resources in Alpha.
  • Cost of delivery team
  • Time to deliver MVP
  • Diversity of user groups tested with
Capacity -
Information on partner participation in building the service.
  • Frequency of code release by delivery team with partners
  • Lead time for changes (time from commit to production)
  • Time to restore service (e.g., from an outage)
  • Code change failure rate (% changes causing failure)
  • % of positions on CDS team matched by member of partner team
  • % of product code available as open source
Performance -
Information on how well prototypes are meeting user needs.
  • Number of critical design issues in most recent iteration
  • Number of accessibility issues in code
  • Comprehension of content by users
  • Reaction to the proposition of the service
Outcome
Information on how the product or service might deliver public benefit.
  • Target increase in awareness of available benefits or services
  • Estimated $ avoided or provided to users due to the uptake of the service
  • Projected time saved for users in applying for the benefit
Beta
Delivery -
Information to understand how well we are using our resources in Beta.
  • Cost of delivery team
  • Time to public launch from partnership agreement
  • Usefulness of product documentation for partner
Capacity -
Information on partner participation in building the service.
  • Frequency of code release by partner-led delivery team
  • Lead time for changes (time from commit to production)
  • Time to restore service (e.g., from an outage)
  • Code change failure rate (% changes causing failure)
  • % of positions on CDS team matched by member of partner team
  • Time to acquire partner-owned SaaS or cloud hosting
Performance -
Connecting user needs to ongoing improvements in the functionality and interactions.
  • Completion rate of key tasks
  • Reading level of content
  • Accessibility compliance
Outcome -
Information on how the product or service might be delivering public benefit.
  • Increased awareness of available benefits or services
  • $ saved or provided to users due to the uptake of the service
  • Time saved for users in applying for the benefit
Live
Delivery -
Information to understand how well we are using our resources in Live.
  • Uptime of product or service
  • Ongoing costs for the partner to support the service
  • Number of support requests
Capacity -
Information on partner participation in iterating the service.
  • Frequency of code release by partner-led delivery team
  • Lead time for changes (time from commit to production)
  • Time to restore service (e.g., from an outage)
  • Code change failure rate (% changes causing failure)
  • Number of subsequent feature iterations undertaken by team
  • Number of other multidisciplinary teams established
Performance -
Connecting user needs to ongoing improvements in the functionality and interactions.
  • Completion rate of key tasks
  • Reading level of content
  • Accessibility compliance
Outcome -
Information on how the product or service is delivering public benefit.
  • Increased awareness of available benefits or services
  • $ saved or provided to users due to the uptake of the service
  • Time saved for users in applying for the benefit

Responsibilities for evaluation

Here are the typical responsibilities on a delivery team for evaluation:

  • Design
    Joining in with setting, capturing and analysing metrics; acting on the evaluation to make design choices
  • Development
    Joining in with setting, capturing and analysing metrics; retrieving data generated by applications infrastructure; acting on the evaluation to make engineering choices
  • Partnerships
    Joining in with setting, capturing and analysing metrics; retrieving baseline data specifically about partner capacity
  • Policy
    Supporting product managers in coordinating the team’s evaluation activities; joining in setting, capturing and analysing metrics, and ensuring that it’s ethical, legal, and aligned to other frameworks
  • Product Management
    Coordinating the team’s evaluation activities; joining in with setting, capturing and analysing metrics; ensuring data is being used to make decisions and engage interested parties
  • Research
    Generating qualitative and quantitative data through research and analysis exercises; documenting findings