Evaluation as a service: assessing stakeholder requests for agile teams

At NHS Digital we’ve recently put in place a new front door process for the national digital channels (NDCs): nhs.uk, NHS App and NHS login.

This process is designed to handle the large volume of requests for the channels that come from national and regional stakeholders. These requests are usually things like:

• adding a large amount of new content to nhs.uk
• integrating a major new service into the NHS App
• adding new core functionality to the NHS App
• changing things on nhs.uk to influence patient journeys in the real world.

In this blog post I’ll describe how one part of the process works: evaluating stakeholder requests for possible inclusion on backlogs.

We like to call these requests ‘opportunities’ because a) there’s no guarantee that we will do the thing that’s being requested, b) our evaluation process can also be used to assess problems identified by product teams as well as stakeholder requests, and c) it’s a positive-sounding word.

How we evaluate opportunities

We have a few core principles for evaluating opportunities:

• We frame the opportunity as a problem to be solved and the benefits of solving that problem. This is to avoid prescribing a solution and to help us understand the value of working on this opportunity compared to another one.
• We include evidence and data, for example how many people the problem affects, the operational and clinical impact, and the user research this is based on.
• We work closely with the requesting stakeholders to understand the opportunity and we produce an ‘opportunity canvas’ (a 1-page executive summary) together.
• We ask our product teams to validate the opportunity canvas to make sure we’re not making undeliverable claims on their behalf.

Several requesting stakeholders we’ve worked with have told us that producing the opportunity canvas has helped them to understand their request better and to identify gaps in their thinking. For example, being more specific about the problem and expected benefits, understanding where evidence might be missing, and identifying interdependencies.

Pipeline we’re part of

Evaluation of an opportunity by PAT (our Prioritisation Assessment Team) is just one part of our front door and pipeline process.

Not all opportunities are evaluated so we have to decide which ones do. We have regular feedback loops with our immediate leadership group and the decision-making board who make the final call on whether to proceed with an opportunity and its priority compared to other opportunities.

Assuming an opportunity passes all those hurdles it is added to the NDC backlog for product teams to pick up at an appropriate time.

A simplified view of the opportunity pipeline.

What about empowered, autonomous product teams?

This model probably doesn’t look like it supports empowered, autonomous product teams: we have a separate team evaluating opportunities and a Board deciding whether or to add something to a backlog.

However, this model is not simply leadership telling product teams what to do.

• Health and care is a complex environment. Everything is interlinked and the NDCs — for all their value and importance — are just 1 part of the digital-and-physical service jigsaw. They do not exist in a vacuum and there needs to be a degree of coordination between them and other parts of the NHS.
• Ultimately, product teams own their own roadmaps. Board decisions influence these but — I like to think — more in the form of strategic direction than outright commands. (Of course, we still get JFDI-style requests that cannot be resisted.)
• Conversations and feedback loops with stakeholders, teams and leaders mean that by the time opportunities make it to product teams, they have already been shaped by a product mindset.
• The board is evaluating 1–3 opportunities each month, and not all opportunities are taken forward. Across the NDCs we have something like 20 product teams. It’s impossible for the board to dictate everything that product teams are working on.
• Evaluating opportunities takes time. The evaluation service offered by my team takes some of this load off product teams and allows them to focus on other things.

Marty Cagan describes a difference between truly empowered product teams and what he calls ‘feature teams’. Our model is probably closer to the latter . I’ll continue to use the phrase ‘product teams’ though.

Outputs we create

Our team’s main output is called an opportunity canvas. This is a 1-page PowerPoint document with spaces to address these things:

• What problem are you trying to solve?
• How is the problem being solved now?
• What are the outcomes and benefits to users and the NHS?
• Who are the users?
• How will you know if you’ve solved the problem?
• Funding and resources
• Urgency and dependencies
• Important context
• Scores for strategic fit, value and readiness (see next section for details)

That’s quite a lot of content for a single screen but our users were very keen on having it all on one page.

The aim of this document is to give decision-makers the information and insights they need to make confident decisions about the prioritisation of this opportunity for the NDCs.

Opportunity scoring

As well as written information, we score each opportunity using 3 separate frameworks.

1. Strategic fit framework

This framework helps us understand the extent to which the opportunity is aligned with our strategic aims. We score it High (10 points), Medium (5), Low (2) or None (0) for the extent to which solving this problem would:

• address health inequalities or equality of access
• improve end-to-end user experience
• give users more control over their health data
• improve uptake of self-care and self-serve
• improve uptake of preventative services
• increase adoption of healthy lifestyles
• align with ministerial commitments or other national strategies.

2. Value framework

This framework helps us score the value of an opportunity — this is a particularly useful when comparing very different opportunities. Again, we score it High (10), Medium (5), Low (2) or None (0) for:

• improving health outcomes
• improving people’s experience of health and care
• increasing efficiency or reducing costs
• the number of people or transactions that would be affected.

3. Readiness framework

This framework is to help us understand how ready an opportunity is to progress, and which aspects will require more work. It uses RAG ratings instead of scores, but having a few red ratings does not mean we won’t go ahead with something — it’s an indication that there are a lot of unknowns or issues to unblock.

The categories of readiness we use are:

• service standards
• operational and service
• policy, clinical and legal
• data and information governance
• technology
• skills and resources.

Download the template

Well done for making it this far. Here’s a generic version of our opportunity canvas and scoring frameworks (PowerPoint).

EDIT 20 Sep 2023: link now points to later version of the opportunity canvas — v1.6 instead of v1.3.

Feel free to adapt and use it in your own context. If you do, please tell me about it (Twitter, Mastodon) — I’d really like to hear what you think.

More to come

I talked about ‘Evaluation as a Service’ at the January 2023 Product People Unconference and there were loads of good questions and comments (see the notes here). I will return to these and other topics in a future blog post.