What would a better COVID test system setup look like?
Multiple files from multiple locations were received by PHE, then processed and consolidated into a single file in order to “upload to NHS Test & Trace team as well as other government computer dashboards,” according to the BBC2. This shouldn’t be a complex process, but this is where the problems began.
Given that PHE were central to the Government’s estimation on 21st September that “cases could reach 50,000 by October,” it’s not unreasonable to expect systems and processes to have been built with this volume in mind, or at the very least, with scalability in mind3.
The biggest issue is that the process design was fundamentally flawed. Any critical process such as this, should have had some error handling and reporting capabilities built into it. James Lewis, Managing Director at Pro2col suggests, “Taking eight days to identify the problem is unreasonable, and unnecessary. There are a number of ways basic error handling mechanisms could have been built into their workflow processes that wouldn’t have impacted performance design.”
- Security controls – Test results are classified as personal, sensitive data.
- Automated onward processing – This might include results alerts, moving to the Test & Trace team, and reporting.
- Reporting / error handling – The governance of sensitive data is critical, even more so in this case.
The CSV files were then presumably processed automatically. The contents of each CSV exported and written to the XLS file. This is where the error most likely occurred, and could have been avoided.
Whilst we don’t know what other steps might have existed in their process, it is possible to select multiple CSV files and merge into a single CSV file. This would offer much larger capacity and ensure that all data is imported.
- A check on the file creation date to ensure that only the correct files were being processed would be the first step.
- Error handling can take many forms. A simple process would be to read the number of lines of data contained in each CSV file, total the number of records across all files, and match that against the output file. The result? A mismatch is quickly identified and action can be taken to resolve it before, it makes an impact. This would also allow any receiving party to check the governance of the data received before delivering it to any onward process.
- Uploading an XLS file to the NHS Test & Trace team would likely result in a more complex routine for extracting the data than if it was provided in CSV format, which is much easier to import into onward systems.