NHS Directory of Services (DoS) Incident Update 2 – Final update

Incident Reference: INC0421098
Service: NHS Directory of Services (DoS)
Priority: P1
Status: Resolved
Update: Final update

Executive Summary

Between 02:00 and 18:11 on 08/09, multiple organisations and services encountered failures when interacting with the Directory of Services (DoS). Affected users experienced intermittent authentication failures impacting critical DoS-dependent services, including NHS 111 Online, 111 Telephony, Ambulance Services, ED Streaming, EPS DoS API consumers and Check Capacity Summary functionality. The incident resulted in varying levels of service degradation across urgent care pathways and associated integrated systems.

Issue
Multiple users and connected organisations were experiencing failures when performing DoS lookups and related API transactions. Reported symptoms include:

·       Authentication failures, including “account locked” messages.

·       Inability to retrieve service information through DoS lookups.

·       Failure of Check Capacity Summary requests.

·       Failure of EPS DoS API searches used to locate pharmacy services.

·       Impact to Streaming & Redirection functionality.

·       Service degradation affecting NHS 111 Online and 111/999 Telephony pathways.

Root Cause
Root cause remains under active investigation at the time of writing.

The leading hypotheses under investigation are:

·       Authentication processing failures affecting API consumers.

·       Request formatting or technology stack inconsistencies between successful and failed requests.

·       Routing or traffic path anomalies causing differing behaviour across consuming organisations/services.

Investigation & Resolution
Investigations have ruled out account lockouts, recent credential changes, infrastructure outages, AWS, DNS and wider platform availability issues. Current evidence indicates REST API services remain unaffected, with the issue pertaining to SOAP API-based integrations used by a number of urgent care systems. Testing has also confirmed affected accounts can successfully authenticate outside of the impacted production paths, suggesting the fault is not related to credentials or password management.

Analysis shows elevated authentication failures across multiple providers, including NHS 111 Online, DHU, LAS, SCAS, Anima and ED Streamer, with some organisations experiencing persistent failures and others intermittent disruption. A rolling restart of underlying service architecture was performed by technical teams at 12:00, however this did not resolve the issue.

Investigations have identified that the trigger for the issues stems from the parsing of the trace parent header, which appears to be causing the failure. Tests of manually removing the header have returned positive results, with submissions completing without error. Analysis into the use of the header has shown that omission of the header will impact some aspects of telemetry reporting against some services, which is not widely used against the impacted services.

An additional potential fix (this involved removing the optional “traceparent” HTTP header from API requests. This header was not used within the Directory of Services, and evidence indicated that its inclusion was contributing to the issues observed) was made available for providers to implement directly if they wished. This information was shared with Providers IT teams, system management teams and system suppliers. Some users were able to work around the issue by changing the way their systems sent requests to the Directory of Services API. The fix was implemented by NHS 111 Online, which subsequently saw successful messages flowing from approximately 17:35 onwards.

Following successful testing, NHS England technical teams implemented the identified mitigation fix at 18:11, restoring service functionality and resolving the issue.