DeepMind Health, the division of the Google-owned AI company that’s applying machine learning to medical data in the hopes of profiting from diagnostic gain, has inked another services agreement with the UK’s National Health Service — expanding the deployment of an alerts, messaging and task management app, Streams, to a hospital in Taunton & Somerset.
This expansion comes despite ongoing controversy over the company’s first NHS data-sharing agreement. The sharing of 1.6 million patients’ medical records with DeepMind by the Royal Free NHS Trust during the development of Streams remains under investigation by the UK’s data protection watchdog, the ICO.
Patients were not informed nor their consent sought. Yet the Streams app has since been actively deployed in the Royal Free’s hospitals. And DeepMind is now forging ahead further, by inking a commercial agreement with a second NHS Trust to deploy the task management app.
Announcing the latest Streams app agreement on its blog, DeepMind makes no mention of the ongoing data-sharing consent controversy attached to the app’s development.
“At Musgrove Park Hospital, part of Taunton and Somerset NHS Foundation Trust, [Streams’] features will alert doctors and nurses to a potential deterioration in their patients’ vital signs that could indicate a serious problem,” it writes. “We believe that by making it as quick and easy as possible for clinicians to intervene if something is wrong, we’ll be able to improve patient safety across the hospital.”
DeepMind would not comment when asked about the ethics of expanding a commercial rollout when doubt has been cast over the legal basis under which patients’ data was obtained and used during the development phase of the app.
DeepMind and the Royal Free previously maintained there was no need for them to obtain patient consent for the sharing of medical records as the Streams app would be used for so-called ‘direct patient care’. However this May a review of the arrangement by the UK’s patient data safety advisory body, the National Data Guardian (NDG), resulted in Dame Fiona Caldicott taking a very different view.
“My considered opinion therefore remains that it would not have been within the reasonable expectation of patients that their records would have been shared for this purpose,” wrote Caldicott in a letter sent to DeepMind and the Trust in question on February 20.
The NDG has been liaising with the ICO as part of its investigation into the data sharing.
Asked whether it has had any specific concerns relating to the controversy surrounding Streams, the Trust’s deputy chief executive Peter Lewis told us: “Clearly we are taking the information governance issues around this very seriously. And we’ve looked at exactly what we are doing with that app, and will be doing with DeepMind. Clearly we’ve taken the appropriate advice on that. So we’re quite clear in terms of what we’ve done and that it’s legal, what we’re doing.”
There is not yet a firm date for launch of Streams within Musgrove Park Hospital, whose website states it treats more than 450,000 patients annually. “We’re starting on the planning phase just now,” Andrew Forrest, CIO, told us. “We’ll be quite open about when we’re going to start using it as a pilot once we know.”
The Trust also said it does not yet know how many patients’ medical records will be shared under the arrangement — because work has not yet commenced and no data has started flowing.
In a statement about the forthcoming deployment of the Streams app at Musgrove Park Hospital, the Trust says the app “will be available at the bedside to alert doctors and nurses to any patients needing immediate assessment, and help them rapidly determine whether the patient has other serious conditions such as acute kidney injury”.
“What we’re doing with DeepMind is not inconsistent around the roadmap we had with digital anyway,” added Lewis, in an interview with TechCrunch. “In terms of thinking about how do we need to run a hospital if we can really take advantage of what we can do digitally — what will make a big difference?”
Data streams under review
DeepMind co-developed Streams between fall 2015 and 2016 with the Royal Free NHS Trust. The app uses an existing NHS algorithm to push alerts to care staff when a patient might be at risk of a condition called Acute Kidney Injury.
However after it emerged how much patient identifiable data had been being shared under the original arrangement, and how regulators had not been pro-actively informed of the project, the ICO initiated a probe.
DeepMind and the Royal Free subsequently went on to reboot the project via a new contract in fall last year. Yet underlying data governance questions remain.
The new services agreement, with Taunton & Somerset NHS Trust, appears, at first glance, to be more a robustly defined entity than the original information sharing agreement (ISA) inked with the Royal Free NHS Trust.
A redacted version of the services agreement between DeepMind and Somerset has been uploaded to DeepMind’s website. Several portions have been entirely redacted — such as a section listing subcontractors on the project, and a diagram showing a data flow map.
Notably, though, there’s a narrowing of the types of data that are being shared — to specify the Somerset Trust “shall not provide DeepMind with access to the personal data of patients who are not active patients”.
This contrasts favorably with the first Royal Free ISA which included sharing historical medical records from patients dating back five years, for example, many of whom may no longer be actively undergoing treatment at the hospital or ever likely to return.
There is also a limit on DeepMind’s processing of Trust patients’ personal data so it cannot be used to develop software outside the remit of the agreement.
And there are what look like terms to purpose-limit how DeepMind can exploit any IP that emerges as a result of the collaboration — i.e. insights that can be deduced from the data, and/or derived as a result of the company being in a position to see how NHS detection moves through to treatment, for example.
Again, IPR limits were missing from the first Royal Free ISA.
Here’s the relevant para from the Somerset agreement:
The Trust’s Lewis told us: “Two things are really important that we have been very clear about: one is that we’re not using any patients data for testing. We will use dummy date for testing the app here. And we will then, as we roll it out, we will be using live data and it will only be our clinical staff who are accessing that live data to treat the patients when we do it.
“The second important consideration was the amount of data that is transferred — so that we’re only transferring data that relates to active patients who are under our care… So it’s very clear therefore what patients are active and only transferring data relating to them.”
He added that “active patients” means patient who are on an “open treatment pathway with us”, and patients admitted in emergencies in some circumstances — given there may also be a risk of A&E admissions developing AKI.
“What we’ve also done is written to the National Data Guardian outlining this approach — not just to seek advice if we are taking the wrong approach but to confirm that we are doing work which is safe and which is legal,” added Forrest. “We haven’t started this work yet — this is our planned approach.”
Giving an early take on the latest Streams agreement, professor Eerke Boiten, who conducts research in data privacy and ethics, suggested that “lots of lessons” have been learned by DeepMind after its Royal Free collaboration ran into trouble — noting, for example, that the Somerset agreement includes an “explicit statement that data will not be linked”.
Concerns have previously been raised about what else the Google-owned company might do with Royal Free patients’ data as a result of a lack of clearly defined limits in the contract.
Digitizing health data and delivering apps
Streams has never utilized any of DeepMind’s AI expertise or its own machine learning algorithms. Although a memorandum of understanding between the company and the Royal Free set out DeepMind’s hope to “gain data for machine learning research” as part of the wider, multi-year collaboration between the pair.
There is no similar MoU with the Taunton & Somerset NHS, according to Lewis. Nor is there any mention of applying AI or DeepMind machine learning algorithms’ to its patients’ data — the focus is on text messaging and clinical task management.
“This is specifically about the Streams application. We don’t have any other agreement — an MoU or a formal contract — with DeepMind. Just this one,” he confirmed, adding that the Trust has no interest in applying AI to patient data at this time.
The five year services agreement with the Taunton & Somerset NHS Trust, which was signed on May 10 this year, covers both the delivery of the Streams app and an underlying patient data API infrastructure — what’s known as an FHIR API — which DeepMind fleshed out in more detail last year.
While the company began its health tech push by working on a single app (Streams), it has since bolted that to this underlying app delivery infrastructure — thereby potentially positioning itself to be a broker for other app developers’ wanting to push their own apps to NHS Trusts in future.
On this possibility the Trust sounded upbeat.
“The whole point of the FHIR development is it’s open and standardized so that there are options which become available. I think the FHIR standard will become the defacto standard across the NHS completely,” said Forrest.
While the services agreement is tied to Streams, the Trust is also evidently hoping to be able to apply a similar task management app approach to additional conditions in future — with the agreement talking about additional “modules” that might be developed for the app, i.e. to expand out from just acting as an earlier detection warning system for AKI.
Lewis explained the aim is to digitize other existing paper-based process — though it’s not yet clear exactly which additional ‘condition alerts’ might be bolted on to its deployment of Streams in future.
“This is really based on the patient, so that what we’ve got in there is data about the patient and about what’s happening to the patient while they’re here with us as an in patient that can help us identify when there are potential problems,” said Lewis.
“So whether it’s about AKI or whether it’s about sepsis, for instance, we will know when things are happening with that patient that we need to react to. And all of that happens at the moment — the problem is a lot of it is recorded manually on paper, and staff have to do specific calculations to calculate what’s called things like ‘early warning scores’, which means we need to react. So whilst that works well on paper you’ll appreciate it’s much better to digitize that and have it done much more in real time, and then the alerts and so on are automated.”
“That’s how we really will drive safer patient care through doing this,” he added.
Commercial question marks
Lewis confirmed to TechCrunch that the Trust did not put out a tender for the development of what it bills as a “patient-focused mobile app”.
Rather the agreement with DeepMind came out of a series of conversations with the company. He noted that the Trust was last year one of a handful named as a “global digital exemplar”, which led it to “various conversations” with suppliers — aimed at delivering on a digital roadmap.
“We were having a conversation with various suppliers, including DeepMind, and this agreement has come out of that,” he told us. “I can’t remember exactly when the initial contact happened but there was quite an informal contact between us and DeepMind and this just grew out of a conversation that was happening… It wasn’t something that we deliberately went and said this is what we need to do.”
What we haven’t done, and haven’t needed to do, because we’ve got legal advice for this, is gone out to a full competitive tender process.
Is that standard practice for the Trust’s procurement of services? “It depends on exactly what we are doing and how we are testing those options. So we didn’t just say ‘DeepMind’s the only option and that’s what we’re going to do’ — we’ve been looking at alternatives. What we haven’t done, and haven’t needed to do, because we’ve got legal advice for this, is gone out to a full competitive tender process.”
He also confirmed there are no wider agreements with the company — such at the Royal Free’s MoU. But he declined to disclose anything about the commercial arrangement wth DeepMind.
“We’re not at liberty to discuss the commercial detail of the contract,” said Lewis.
Commercial terms have been redacted from the ISA uploaded to DeepMind’s website:
On this, a DeepMind spokeswoman told us: “The contract is minimally redacted… I believe it would be up to the trust if they wish to disclose any further commercial information.”
“Regarding the business model, under our agreement the Trust will pay DeepMind a limited service fee if DeepMind incurs support costs when providing the services. As with our other agreements, we don’t believe in charging the NHS for yet-to-be-proven solutions,” she added.
“Only when we can prove that we have improved outcomes will we be paid accordingly within IT supplier market rates. We’re not driven by a desire to maximise profit, but rather to create a mutually sustainable business model so that we can continue to grow our team, work with more hospitals and, ultimately, help more patients.”
The ongoing lack of any commercial information for the pricing of the Streams service is significant given that this is a service being sold to the public sector. And if NHS Trusts are inking deals with a tech giant without putting out open tenders there could be legal issues to consider — such as compliance with procurement rules, and even concerns relating to possible anticompetitive behavior.
Although evidently Lewis is confident that the Trust’s own legal advice on not needing to run an open public procurement in this case is sound.
Another consideration: The BBC reported earlier that NHS patients are not being offered an opt-out for their medical records to be shared with the Google-owned company.
Asked about this, Lewis said their aim is to engage with patients so they are aware of the app and its implications.
“We need to work this through. Clearly there’s a balance here — we’re trying to be very open about what we’re doing and therefore we have a process in place from here on about how we’re engaging with patients to work through exactly how we do that. But this is built on the premise that we are using the information about these patients in this way to improve their care and make them more safe so that particularly we’re escalating deteriorating patients much earlier and we can therefore react to that,” he said.
“That’s the whole premise on which we’re doing this but we are going to go through that engagement process with patients — one so they can understand exactly what we’re doing, and two so that we can understand those concerns and look at how we can mitigate them.”
The Trust will be holding workshops, displays and open day events with staff and the public in the coming “days and weeks”, so that people can “see how the app works, what it will mean for patients, and how it might be developed in future”, as it puts it.
The first of these events will take place on July 17, with information slated to be displayed in the public concourse of the hospital.
The Trust has also indicated it intends to publish its Privacy Impact Assessment of the agreement with DeepMind — though at the time of writing this document has not yet surfaced.
In a statement about its ongoing investigation into the original data-sharing deal between DeepMind and the Royal Free, an ICO spokesperson told us: “The ICO continues to investigate the sharing of 1.6 million patient details by the Royal Free NHS Foundation Trust with Google DeepMind in support of the testing and development of an alert, diagnosis and detection system for acute kidney injury. We hope to conclude that work shortly.”
“We will also speak with Taunton and Somerset NHS Foundation Trust to offer our advice on the law,” it added.