How Imperial College London is accelerating dementia research with a modern data platform

Imagine being unable to tell your doctor whether you’re in pain or running a fever. This is a reality for many people living with dementia — and it means doctors can struggle to make the right diagnosis, leading to delayed treatment.

For people living with dementia, subtle changes such as sleep disruption, reduced movement and shifts in daily routine can signal meaningful changes in health. But when people living with dementia aren’t able to fill in the gaps themselves, capturing that data and making it useful for care providers can significantly improve outcomes. At the UK Dementia Research Institute Centre for Care Research and Technology (CR&T), based at Imperial College London, researchers track these signals continuously. Using data from in-home sensors, sleep monitors, and electronic health records, the team builds a real-time picture of the person’s health to improve care and advance research. This picture can pick up infection early, help reduce avoidable hospitalisations, and help people live safely at home for longer.

But over the years, as the number of homes, in-home devices, and data volumes grew, the data platform behind that mission struggled to scale at the same pace, creating challenges for delivering timely, reliable insights to support care and research.

When critical data can’t move fast enough

Over five years, the CR&T’s flagship service, the Minder platform, evolved into a rich infrastructure, although the platform’s growth brought with it increasing challenges around scaling.

As data volumes grew and use cases expanded, three challenges began to slow progress:

1. Competing workloads slowed innovation – Systems handling ingestion, analytics and real-time queries began to overlap. Even small changes risked breaking production workflows, forcing teams to move cautiously and slowing iteration.

2. Storage and compute were tightly coupled – To keep data accessible, large volumes were stored in operational databases. As data grew, so did infrastructure costs, with no clear path to scale efficiently.

3. Researchers couldn’t easily access data – There was no dedicated research environment. Non-technical stakeholders, including clinicians, had limited visibility into the data, making it harder to validate models and translate insights into care.

These issues delayed the translation of the Centre’s research to clinical practice.

Building a platform designed for research and care

To move faster, the CR&T re-architected its platform with the goal of separating systems that had previously been tightly coupled and creating a dedicated environment for analytics and research.

IoT data is now ingested and validated through a Kubernetes layer before landing in Delta Lake on Azure Data Lake Storage. Data progresses from raw (bronze) to refined (silver) to anonymized, research-ready datasets (gold), which power downstream analytics.

This shift created a modularized, reliable, and scalable foundation for working with continuously growing sensor data, all without impacting operational systems.

At the same time, the CR&T preserved what already worked for clinical workflows while modernizing everything around it. EHR systems remained optimized for interoperability with NHS and other clinical environments, continuing to use the FHIR standard to ensure seamless data exchange. This foundation is now enabling active integration with NHS clinical care via Imperial College Healthcare NHS Trust, bringing Minder insights closer to frontline decision-making. Early deployments are focused on embedding remote monitoring data into clinical workflows, supporting clinicians with more timely and contextual information about patients living at home.

On top of that foundation, the team introduced centralized governance through Unity Catalog (UC), enabling fine-grained access control across research teams, studies and external collaborators. Databricks then became the dedicated analytics layer, giving researchers a unified environment to explore data, build models and collaborate independently of production workflows.

For model deployment, the CR&T continues to use Kubeflow, while actively evaluating MLflow to further streamline experimentation, deployment, re-training and maintenance of models.

Turning data access into research velocity

Modernising the architecture was only part of the solution. The CR&T also rethought how researchers interact with data, building a research-to-production workflow that accelerates how insights are developed and shared. Unity Catalog plays a central role by tracking dataset usage and helping identify high-value data assets. Analytical and processing pipelines developed by research teams on frequently used datasets are code-hardened and made reusable across teams. This reduces duplicated effort and accelerates delivery by giving researchers gold-standard pipeline templates for working with new or complex datasets.

Accessibility also improved significantly for clinicians and other non-technical stakeholders. Databricks dashboards now surface IoT device health, behavioural and physiological trends, and cohort-level insights in a more intuitive way. Furthermore, embedded dashboard integrations are being tested within monitoring systems so that clinicians can access insights directly within the tools they already use.

The platform also addresses a critical requirement in medical research around reproducibility. IoT data updates continuously, so results can change over time. To ensure consistency, every data point is stored with its original timestamp, allowing researchers to reconstruct exactly what a clinician saw at any point in the past.

From months to weeks—a real impact on productivity

By building the new platform alongside existing systems, the CR&T avoided disruption while accelerating progress. Early results show meaningful gains:

• 100% uptime maintained throughout the migration
• New IoT data sources integrated in as little as one month, down from ~6 months
• Model development reduced to ~1 month, enabling faster iteration
• Rapid data growth, including millions of IoT data points ingested within months
• 50% month-over-month platform growth, with rising adoption among non-technical users

Most importantly, these improvements are translating into real-world impact:

“We’ve restructured how we work and made data more accessible. The Databricks analytical platform has already made clinical insights available for 581 people living with dementia in the last 5 months.”—Ethan de Villiers, Data Engineer, CR&T

The team also estimates saving hundreds of engineering hours compared to building equivalent infrastructure from scratch.

Advancing the mission for better dementia care.

At the CR&T, the work is ongoing. For a population that often cannot advocate for itself, the ability to surface objective, continuous data about what is happening at home is a core part of delivering care. As the platform grows, so does the potential to reach more people, compress the time between a research insight and a clinical decision, and give care teams the evidence they need to act.

The CR&T’s experience also reveals that the biggest barrier to data-driven care is rarely the data itself. It’s whether the right people, regardless of their technical knowledge, can access it, trust it, and use it. That’s the problem the CR&T set out to solve. And the data suggests it’s working.

Lessons for building healthcare data platforms

The CR&T’s experience reflects a broader shift happening across healthcare, where the future of care depends on turning fragmented, real-world data into actionable insight.

As organisations increasingly adopt connected devices, remote monitoring, and AI-driven analytics, the challenge is no longer simply collecting data. It’s building systems that make that data accessible, trustworthy, and usable by the people making care decisions every day.

For dementia care especially, where people may not always be able to communicate changes in their condition, continuous data can provide critical context that would otherwise be missed. The impact extends far beyond a single use case. The same architectural principles around scalable data infrastructure, governed access, and researcher-friendly analytics, are becoming foundational for modern healthcare systems seeking to accelerate research, personalize care, and improve outcomes at scale.

The CR&T’s work demonstrates how a shared, trusted data platform can help healthcare organizations accelerate research, improve clinical decision making, and ultimately, deliver better patient outcomes.

Acknowledgments

We acknowledge the members of the core team at Care Research and Technology Centre, our funders and study sponsors for supporting this work. Special thanks to the Data Science and Software Teams (Nora Joby, Anna Joffe, Ethan de Villiers, Amer Marzuki, Ramsheed Abdul Rahim and Gaia Frigerio) for their technical contributions in developing this platform.

Funding & Support

Minder is supported by the UK Dementia Research Institute (UK DRI Ltd), which is principally funded by the UK Medical Research Council, with additional support from the Alzheimer’s Society. Mindercare is similarly supported by the UK Dementia Research Institute (UK DRI Ltd), principally funded by the Medical Research Council, with additional funding from LifeArc.

Learn more

Get regular updates about how Databricks helps public sector organizations unify data, govern AI and turn knowledge into action at global scale by following Databricks for Public Sector on LinkedIn.