The new Publications system replaces the existing statistical publications and clinical indicators systems which pre-date NHS Digital.
There are some important security and availability requirements for the system:
- Publishing time must be within 59 seconds of a specified time
- Pre-release access to the data is strictly controlled
The users of this service fall into a few common groups, such as: health and care professionals, researchers, commissioners, statisticians and journalists.
Department / Agency:
NHS Digital
Date of Assessment:
27 June 2017
Assessment Stage:
Alpha
Result of Assessment:
Not pass
Lead Assessor:
S.Dixon
Service Manager:
R.Donald
Digital Leader:
G.Ayling
Assessment Report
Outcome of service assessment
After careful consideration, the panel have concluded that the service did not meet the standard because:
- The expected outcomes of the alpha phase have not been completed. The team are doing good work but we do not think they are at the end of an alpha phase yet. There is still work to do and there are a number of gaps, including testing the full end to end journey and the technical prototypes, as well as putting in place a full team.
- They have not sufficiently demonstrated how the research has informed the design or iteration of a solution.
Service assessment
User needs
We were very impressed by the product owner's enthusiasm for their users - he mentioned that he has listened in to contact centre phone calls and has encouraged senior management to do the same which is really great. They demonstrated a good understanding of who the users were, identified using a range of sources, as well as a variety of needs at various levels of granularity. However, the categorisation of users and their needs was driven by how they accessed the service rather than by what they were trying to achieve.
The user needs presented were at a very detailed level, presupposing their existing service, rather than at a high level based on users motivations, emotions or goals. When prompted the team was able to demonstrate an understanding of what these high level needs were and that they could be related to the more detailed needs.
While some good research has been done, research on the full end to end journey is needed, as well as research focusing on users with accessibility or support needs. There has been team involvement in research and sharing of the research findings and project amongst stakeholders. It would have been helpful to see some of the products of research, such as personas or journey maps, in order to support their understanding other the service users.
Clearly the team is thinking about what their users need and have done good work mapping out their routes and pain points though it has not been demonstrated adequately how this research has informed the design or iteration of a solution.
Furthermore without a fully functioning prototype having been tested it is unclear how they would be able to start building in the next phase of work.
It does appear that adequate resource is now in place to move forward with prototype and usability testing including a designer and full time researcher. The team appear to have a plan to begin exploring the full user journey and focus on what their users are trying to achieve as well as to iterate this design. However, much of this is activity that should have been conducted already and should not be conducted alongside the service build as it may well be constrained by its requirements.
Team
They clearly explained their team structure and have a good understanding of the roles required. However, the team doesn't include any technical roles except for the technical/solutions architect.
They're using the Digital Outcomes and Specialists framework to try and get a development team but haven't been successful yet. We were encouraged by their aim of using interim staff to begin with and hiring permanent staff into the team in the future. It's good that the technical team will be located with the rest of the team rather than being remote.
The team would really benefit from a developer and designer working closely with their user researcher to create rich prototypes more iteratively and put them in front of users.
They are working in an agile way, currently working in Kanban and will be moving to Scrum. They already do stand ups and have put in place proper tools by using Jira manage sprints. They demonstrated a good understanding of the agile process and ways of working. We would encourage them to start other agile ceremonies (e.g. sprint planning and retrospectives) as soon as possible.
The team are already co-located and any technical resource they bring in will also be in the same place. It is good to see they do not want a geographically distributed team.
Technology
The team have not built much yet but the work they have done so far is good for the beginning of an alpha.
We were impressed by their enthusiasm for hosting the service using a public cloud and would encourage them to push as hard as they can with the information governance team at NHS Digital to make sure that happens.
We were also impressed by their commitment to avoiding vendor lock-in and the work they've done so far using open source products. Also with the modular architecture approach, however we would be keen for them to use this despite any time constraints.
They're building on top of existing products and services which is great and are looking at cross government platforms. They should be able to sign up for a test account for GOV.UK Notify now at https://www.notifications.service.gov.uk/
We would encourage them to focus at first on building a basic service that meets user needs and improve it once it's running. For example, they might be able to deprioritise:
- an improved search backend until they're sure that their CMS doesn't provide one which is good enough
- user-generated content
The plan to test the service with real users to decide on a CMS is good. The "product bake-off" idea they have planned to evaluate a couple of different solutions would make a good end to their alpha and would inform what they want to build in beta. This would allow them to build on the winning prototype and keep iterating. However, we would have expected this to have been completed before the alpha service assessment.
They are currently constrained by the content management system of the NHS Digital website but expressed a desire to make their code open source and well documented.
We were impressed with their approach to security and risk management, including considerations of information governance and the privacy impact of the service.
Downtime and service recovery is being planned as part of the ongoing service management. They have a disaster recovery plan currently but it needs improvement.
Design
The team has not yet demonstrated an ability to design and test the end to end user journey utilising a prototype similar to the real world user interface. While this is obviously impacted by the availability of resources this sort of testing and iteration would be essential in order to progress to building a service.
As the team has not been able to test and iterate a prototype that encompasses an end to end user journey it is difficult to assess the effectiveness of their proposed solution.
They have considered using the contact centre to promote digital take up and for users who are unable to access the data online. It is good to see they are considering the effectiveness and impact of other channels including social media and news releases to increase uptake of the service.
As an arm’s length body, they are not able to test the service with the minister but they are testing it with senior people in their organisation.
Analytics
Due to the nature of the project, the team considered the GDS KPIs are decided they were not appropriate. However, we would encourage them to do some thinking about the cost user for the service.
They have undertaken some good analysis of the service data and compared it to other websites. It is good to see they are looking at user journeys and traffic on the site in order to set up funnels and measure impact.
They have defined a set of baselines which they are measuring but we would encourage them to define KPIs which are aligned with high level user needs rather than the granular needs they have currently.
They do not send any data to the performance platform.
Recommendations
To pass the next assessment, the service team must:
- Have defined higher level user needs which are distinct from the current product specific ones. They are included in the work done so far but need articulating differently. They should also consider doing some personas for users based on what they are trying to achieve.
- Have developed and tested a fully functioning prototype of the whole end to end journey, as well as research focusing on users with accessibility or support needs.
- Have in place the technical team required and have completed the ‘bake off’ of CMS solutions.
The service team should also:
- Continue to improve their agile ways of working.
- Be able to clearly explain what parts of the overall user need is met by other organisations or services, and how the coordination and hand-off between the service and the organisations work.
- Do more to understand how users who need support will move between digital and the contact centre, to make sure that support routes are clearly signposted and easy to access.
The panel would like to thank the service team for their time and the effort the team went to in preparing for the assessment. We look forward to seeing further progress in the months ahead.