From: DHSC
Assessment date: 20/02/2024
Stage: Alpha
Result: Met
Service provider: NHS England
Service description
Cohorting as a Service is an assured way for national healthcare teams to identify patients for care, based on risk factors.
The service includes a managed service for creating, assuring and running and disseminating cohorts, underpinned by a rules engine (Cohort Processing System). It also includes a digital journey for finding out why a patient is included in a cohort. The service also includes publishing cohort information to the web.
Service users
- Requestors: People delivering national health programmes who want to identify patients for care
- Creators and Facilitators: NHS England staff creating and delivering cohorts, including clinical experts
- Enquirers: People looking for information about cohorts, including Live service teams, patients and healthcare professionals
Report contents
- Understand users and their needs
- Solve a whole problem for users
- Provide a joined-up experience across all channels
- Make the service simple to use
- Make sure everyone can use the service
- Have a multidisciplinary team
- Use agile ways of working
- Iterate and improve frequently
- Create a secure service which protects users’ privacy
- Define what success looks like and publish performance data
- Choose the right tools and technology
- Make new source code open
- Use and contribute to open standards, common components and patterns
- Operate a reliable service
1. Understand users and their needs
Decision
The service met point 1 of the Standard.
What the team has done well
The panel was impressed that:
- the user research presented was of high standard and demonstrated an understanding of the breadth of the impact of cohorting as a service to support public health intervention
- the presentation and outputs clearly reflected the testing of the riskiest assumptions made following Discovery
- the team had investigated the spectrum of activities, from requests coming in, through the processes undertaken by internal actors (as both service providers and system users), to the handling of enquiries generated by data subjects
- user research had helped to define the Core users of the service and their collective needs
What the team needs to explore
Before their next assessment, the team needs to:
- explore how the grouping of users’ roles within one persona (especially Facilitators and Enquirers) might affect their interaction with or experience of CaaS
- consider how separate personas, not least Patients and GPs/practice staff might avoid unintended consequences of the cohorting process, including investigating routes in to the “find out why…” service
- break down the current personas into more detail. The current personas more reflective of roles within the system, but within these roles there will be a variety of needs, contexts and motivations
2. Solve a whole problem for users
Decision
The service met point 2 of the Standard.
What the team has done well
The panel was impressed that:
- the team have involved clinical users and have prioritised clinical safety and data quality based on feedback. This led them to disprove one of their hypotheses early on
- the team have mapped the backend business and clinical processes, and a high-level journey with offline and online touchpoints
- the team have researched the time taken to manage enquiries and causes of delays in the as-is service. They plan to introduce changes that will open the cohort search and reduce communication cost
- the team explained well the concept of CaaS as a component (analogous to Notify) that can be used within other services
What the team needs to explore
Before their next assessment, the team needs to:
- work in the open
- consider a more proactive approach to managing awareness of the service at the start of the user journey
- work on the longer-term scalability of the service; what would be the impact on internal staff and external users if the service was scaled to meet all the potential cost savings for the NHS?
- consider the scope and focus of what is achievable in private beta and remember what the core problem is they’re trying to solve. It is good that the team are thinking about the journey end-to-end, but there is a risk the team could spend private beta on a solution in a journey stage for the enquirers that isn’t appearing to solve the core problem they identified
3. Provide a joined-up experience across all channels
Decision
The service met point 3 of the Standard.
What the team has done well
The panel was impressed that:
- the research included content and services the user interacts with outside of the current diabetic eye screening service, such as the immunisations green book
- the team have spent time mapping out and helping to iterate the process flows in the as-is journey, looking at where technology can streamline interactions and activities
What the team needs to explore
Before their next assessment, the team needs to:
- look for ways to reduce or satisfy queries that currently manifest as questions being posed to professionals outside CaaS, as well as enquiries coming into the service, due to inclusion in or exclusion from cohorts (catered for in the Find out why part of the service)
4. Make the service simple to use
Decision
The service met point 4 of the Standard.
What the team has done well
The panel was impressed that:
- the team have tested and co-created most parts of the end-to-end service that the user interacts with, including online parts and offline parts
- the team have recognised the importance of the human interaction for Q&A fact finding at the requesting stage of the user journey to achieve correct definitions of intent and selection of data sets
- the team have focussed on clinical safety as a necessity
- the team have co-created clinical process and business process maps with users
- all testing of digital interactions where relevant was done in the NHS prototyping tool kit
What the team needs to explore
Before their next assessment, the team needs to:
- create a detailed service blueprint before testing in private beta (for example processes, technology, user steps, touchpoints, and policy if relevant) demonstrating how the journey will work end to end for the frontend and the backend experience at each stage for all users. It is good that the team co-created and tested the backend processes and high-level journey maps with users, but they risk designing a siloed journey if they do not consider how the frontend and backend experience influence one another and how that may impact delivery. Furthermore, the team needs to have a more detailed plan of what the frontend touchpoints are for the requestor. For example, how the requestor is communicated to throughout the journey, and the team should further consider the awareness stage for requestors who are currently made aware of the service through word of mouth. The team should consider how might the service be more proactive with touchpoints, for example with campaigns and emails.
5. Make sure everyone can use the service
Decision
The service met point 5 of the Standard.
What the team has done well
The panel was impressed that:
- the team tested with 7 users who had access needs including cognitive, visual and motor impairments
- the team plan to run a full accessibility audit in beta once production code is running
- the team plan to run a universal barriers evaluation in private beta
What the team needs to explore
Before their next assessment, the team needs to:
- the team must conduct accessibility testing beyond proxy, by testing with people of various accessibility needs across all user groups in private beta, including the user group of creators and facilitators that are yet to test with. It's good to see the team did some initial accessibility testing and have plans for private Beta including testing via proxy
6. Have a multidisciplinary team
Decision
The service met point 6 of the Standard.
What the team has done well
The panel was impressed that:
- there are a mix of roles presented at the assessment and the team members appeared to have an equal voice in the team
- operational and clinical users were involved with design activities
What the team needs to explore
Before their next assessment, the team needs to:
- have a robust plan in place to manage knowledge sharing and make sure roles with a higher level of decision-making responsibility are in-house. The team recognises they rely heavily on external suppliers
- recruit more UCD roles to serve the three teams. There are currently two user researchers, down from four
7. Use agile ways of working
Decision
The service met point 7 of the Standard.
What the team has done well
The panel was impressed that:
- clinical users were brought into design process early on and helped disprove a hypothesis
- senior decision makers in the programme were able to change decisions based on user research, and the escalation route seems clear
What the team needs to explore
Before their next assessment, the team needs to:
- link up with the NHS and cross-government communities of practice to share their work and invite feedback. This can be done via slack channels, open show and tells, blogging and other method
8. Iterate and improve frequently
Decision
The service met point 8 of the Standard.
What the team has done well
The panel was impressed that:
- the team were confident to throw away and iterate ideas in alpha
9. Create a secure service which protects users’ privacy
Decision
The service met point 9 of the Standard.
What the team has done well
The panel was impressed that:
- the team was actively considering security, including threat assessment activities
- the team was relying on existing NHS identity management processes, building on the existing capabilities
- the team actively reviews the application-level security of their code, including reviewing security of 3rd party packages
What the team needs to explore
Before their next assessment, the team needs to:
- improve their development lifecycle to include regular reviews and uplift of the service’s threat model
- develop and test a business continuity plan to have a coordinated response to a major cyber incident or data breach
10. Define what success looks like and publish performance data
Decision
The service met point 10 of the Standard.
What the team has done well
The panel was impressed that:
- the team have used a framework to define their priority goal and developed KPIs around this
- KPIs have been mapped to user needs
- there is a plan for how most metrics will be tracked
- the team have researched current costs of cohort provision, which can be very high
What the team needs to explore
Before their next assessment, the team needs to:
- have a plan for how they will track metrics of enquiries being solved first time
- have a clearer commercial model and ROI (return on investment) calculations to understand in more detail how this service will save money
- explore whether tracking ‘learnability’ is worthwhile measure of success. This is an aspect of usability that is particularly relevant to frequent or ‘super’ users of a service and considers the design trade-offs that might need to be made in a complex internal service
- have a better understanding of how many users are expected after private beta and what and whether their private beta cohort is a sufficient number
11. Choose the right tools and technology
Decision
The service met point 11 of the Standard.
What the team has done well
The panel was impressed that:
- the team have made well-considered choices when deciding to build versus re-use. Specifically, the arguments to build the cohort processing service are compelling and well-made
- the team decided to focus on process optimisation for cohort creation, rather than building technology is also noteworthy, as a choice to not use technology
- the team have made their technology choices in consultation with the wider programme space they operate within
What the team needs to explore
Before their next assessment, the team needs to:
- decide whether the cohort explorer needs to exist as a stand-alone tool or is better served being APIs to integrate into other existing user interfaces
- consider whether terminology tooling (Ontoserver) is an alternative for use of Excel spreadsheets, including promoting easier re-use of terminology code-sets
12. Make new source code open
Decision
The service met point 12 of the Standard.
What the team has done well
The panel was impressed that:
- the team has understood the NHS England quality expectations before open sourcing of code
- the team has been doing the preparation work to open source their code
What the team needs to explore
Before their next assessment, the team needs to:
- open source the software code projects used to build this service
- discuss with stakeholders how to apply the open sourcing principles to the clinical code-sets
13. Use and contribute to open standards, common components and patterns
Decision
The service met point 13 of the Standard.
What the team has done well
The panel was impressed that:
- the team are using OIDC as their identity standard
- the team are relying on previously built shared components (MESH)
What the team needs to explore
Before their next assessment, the team needs to:
- consider how the work done to date might be adopted as a standard methodology for generating cohorts across the NHS and Care systems
14. Operate a reliable service
Decision
The service met point 14 of the Standard.
What the team has done well
The panel was impressed that:
- the team has a process for handling regular Snomed updates
- the team have a testing and assurance process for new cohort rule sets
What the team needs to explore
Before their next assessment, the team needs to:
- have defined and agreed the overall service level of the CaaS service
- decide how to apply versioning to the cohort outputs, recognising the rule and source data versions in that
- have defined and tested a breach response plan, as part of the business continuity planning
- test in private Beta with a manageable number of users, to ensure they are not overwhelming themselves to have time to deal with things that go wrong and have capacity to make changes to the journey for the frontend and backend experience through learnings discovered in private Beta.
Leave a comment