From experiment to insight: how Dotmatics Luma and Databricks make AI-ready science a reality

The gap between scientific data and scientific insight

Modern scientific workflows generate data on an extraordinary scale. A single organization might run hundreds of instruments across wet labs and partner networks. Each produces data, and most of the time that data lives in silos, disconnected from the very decisions it’s meant to inform.

The problem isn’t volume, but rather context. Maintaining the integrity and context of scientific data as it moves across instruments, analyses, and decisions is crucial. When context is lost, scientists spend time reconstructing or repeating results instead of advancing research. When AI models are trained on fragmented, unharmonized data, the outputs can’t always be trusted (Figure 1).

Figure 1. Dotmatics Luma and Databricks transform fragmented instrument outputs into a continuous, connected pipeline of structured, AI-ready scientific data.

Closing that gap requires two things working in concert. A platform purpose-built for scientific data, and the enterprise-grade infrastructure to support it at scale. That’s what Luma, Dotmatics’ scientific intelligence platform, and Databricks were each respectively built to do. Together, they deliver something neither can provide alone.

What Dotmatics Luma and Databricks deliver together

Luma is the scientific operating layer for modern R&D. Luma captures instrument outputs continuously and automatically, without disrupting existing workflows, bringing data into a harmonized, structured scientific record in real time. It can also handle billions of scientific data points daily.

That harmonization step is what makes everything downstream possible. Unstructured raw outputs become structured, FAIR-compliant (Findable, Accessible, Interoperable, and Reusable) data that is ready for analysis, modeling, and AI applications the moment it arrives. Because the scientific record is continuous and structured, AI can be applied to the entire record, identifying patterns across experiments, suggesting what to run next, and even generating plain-language standard operating procedures (SOPs) that scientists can follow immediately.

Databricks is the foundation upon which Luma is built. This provides the scalable, governed infrastructure needed to store, manage, and activate that data across the enterprise. It allows scientific data to sit alongside finance, procurement, and business intelligence systems, connecting research outcomes to the broader organizational context. Delta Sharing enables seamless data exchange with third-party collaborators including contract research organizations (CROs) and academic partners, without compromising governance or data integrity.

How Dotmatics Luma and Databricks work better together

Luma is purpose-built for science, and Databricks is purpose-built for scalable data and AI.  Luma runs natively on Databricks, so organizations get deep scientific capability and enterprise-grade data infrastructure as a unified stack, not a patchwork of integrations. That unified stack works because each platform contributes something the other doesn’t.

Complementary by design. Luma provides the instrument connectivity, harmonization logic, scientific context, and a FAIR-compliant data foundation, all built specifically for R&D. The use of open, extensible ecosystems for both biology and chemistry ensures that users are leveraging workflows designed by scientists, for scientists. Databricks brings the data and AI infrastructure, with scalable storage, governance, and the tools to activate that data across the enterprise. Together, the stack is greater than the sum of its parts (Figure 2).

Figure 2. Luma and Databricks form a unified stack, with scientific capability on top, enterprise data infrastructure beneath, and AI-ready insight as the output.

The result is a faster path to AI-ready science, without sacrificing the rigor that science demands. Luma is built for workflows where data must be auditable, decisions must be traceable, and AI outputs must hold up under scrutiny, spanning everything from early discovery through regulatory submission. That’s the standard this partnership is built to meet.

An example application area: When chromatography data fragments

The typical chromatography workflow is full of operational drag. SOPs can vary across teams and sites, instruments often come from different vendors with their own proprietary data systems and file types, and results are manually exported, reformatted, and loaded into an electronic lab notebook (ELN). This approach can strip out metadata, lineage, and experimental context resulting in difficult cross-site comparisons and underlying data that often ends up buried or inaccessible.

This type of siloed data is precisely what we want to avoid when working in an AI environment. Scientific continuity is crucial, and Luma enables this by acting as the orchestration layer that enables faster scientific decisions with continuity across the entire research lifecycle. This includes:

• experiment design
• automated job delivery and data acquisition
• automated chromatogram analysis through connectivity with powerful tools
• one-click reporting and sharing
• easy data comparison

Importantly, the metadata, lineage, and experimental context are preserved throughout the digital thread.

This is where Virscidian’s Analytical Studio comes in. In 2024, Dotmatics purchased Virscidian, which owns the powerful Analytical Studio chromatography processing software. On its own, this software provides tremendous potential for accelerating drug discovery, due to its capabilities in automating complex liquid chromatography–mass spectrometry (LC/MS) data processing, high-throughput experimentation (HTE), and purification workflows. What might take weeks to do manually can be done in a matter of minutes. By operating in tandem with Luma, Virscidian’s software now gains a results dashboard, compound registration, and compound management tools built into Luma.

Chromatography is just one example of a much broader pattern. The same fragmentation can be observed anywhere instruments, teams, and data formats multiply, whether it be mass spectrometry, plate-based assays, sequencing, imaging, and beyond. Whatever the modality, the underlying problem is the same; context gets lost between capture and decision. Thankfully, the fix remains consistent. A continuous, harmonized record that travels with the data instead of stopping at the point of collection. That’s the value Luma and Databricks deliver across the research lifecycle, not just in one workflow.

Seeing it in practice

A large global pharmaceutical company faced a challenge familiar to any organization running research at scale: more than 5,000 instruments across their campus, each generating data in isolation. Their largest and most fragmented data source was their (LC/MS) fleet, which featured instruments from four different vendors, each storing chromatography data inside its own proprietary system. This meant there was no way to trend performance, compare results across sites, or apply AI to a data set that had never been unified. They deployed Luma starting with approximately 1,500 instruments, connecting outputs from all four vendor systems into a harmonized, FAIR-aligned record without disrupting a single workflow. Scientists continued working exactly as before, except their data no longer stopped at the boundary of each vendor’s system.

For the first time, the organization could trend instrument performance across vendors, run purity analysis from a unified view, and draw on utilization and uptime data to inform capital planning and service contracts. These decisions previously required significant manual effort to piece together from disconnected sources. With a clean, structured, historically complete data set now in place, the organization also gained a foundation ready for AI and machine learning, with a clear path to connecting all 5,000+ instruments across their campus.

What this organization built is not a one-time integration project. It is a repeatable foundation: start where the data pain is most acute, prove the value quickly, and expand from infrastructure that works. That is the model Luma and Databricks are built to support.

Interested in seeing what Luma and Databricks can do for your organization? Visit dotmatics.com to learn more about Luma and the Dotmatics platform.