How Unity Catalog managed tables bring interoperability, performance, and unified governance to the Lakehouse

Unity Catalog was designed for interoperability at scale. Enterprises have the flexibility to run any engine on a single copy of data with governance enforced centrally.

Today, we are taking interoperability a step further. We’re excited to announce that external access to Unity Catalog (UC) managed Delta tables is now in Public Preview.

Over the past year, a growing list of engines, from Apache Spark and Flink to now Starburst and DuckDB, have integrated with UC managed Delta tables, and the ecosystem continues to grow. Enterprises can get the best of both worlds: the price-performance benefits of UC managed tables and the flexibility to use the engines that fit each workload. Unity Catalog remains the central governance layer, ensuring that access policies are enforced consistently across engines.

Until now, teams that needed multi-engine access used external tables, which lack performance optimizations and governance guarantees. With external access now in Public Preview, that trade-off is gone. UC managed tables are the clear choice for the lakehouse across price, performance, and interoperability.

Why teams choose Unity Catalog managed tables

UC managed tables combine the interoperability of Delta Lake and Apache Iceberg with automatic, built-in capabilities that improve query performance, lower storage costs, and keep tables healthy over time. They are also the foundation for platform features, such as disaster recovery and Zerobus ingestion.

Databricks automatically tunes managed tables through Predictive Optimization. Predictive Optimization cleans up storage, collects query statistics, and automatically selects Liquid clustering columns to evolve table layouts as query patterns change. Together, these benefits can deliver up to 50% storage cost savings and 20x faster queries while eliminating the need for data teams to tune tables manually.

We recently announced the General Availability of catalog commits, which make Unity Catalog the commit coordinator for UC managed tables. Rather than having external clients commit directly against cloud storage, writes to UC managed Delta tables are coordinated through the catalog. This makes Unity Catalog the source of truth for table state, which unlocks key benefits like safe external writes, multi-statement transactions, and auditing of external operations.

Create, read, and write from external engines

Now in Public Preview, external engines can:

• Create, read, and write to UC managed Delta tables with catalog commits from Apache Spark (batch and streaming), Apache Flink, Starburst, DuckDB, and Streamnative.
• Keep governance centralized in Unity Catalog across all engines.
• Benefit from Predictive Optimization, which lowers costs and improves performance across all engines.

External access is built on Open APIs for metadata, credential vending, and commit coordination. Credential vending issues short-lived, scoped credentials based on UC privileges, while the Unity Catalog Delta API gives Delta engines a versioned contract for create, read, and write operations. Because these APIs are open and implemented in the Unity Catalog OSS project, any engine can integrate with create, read, and write support. The result is a growing ecosystem of engines that operate on the same governed copy of data in Databricks UC and in UC OSS

Partners are already building around these APIs in different ways. Starburst, for example, integrates with Unity Catalog and Delta Lake to support creating, reading, and writing to UC managed Delta tables.

“Starburst shares Databricks’ vision of openness and interoperability across the data ecosystem. By integrating with Unity Catalog managed tables, we are enabling customers to create a single source of truth for all their data, with centralized governance and the flexibility to leverage the tools of their choice.” —Justin Borgman, CEO, Starburst Data

Expanding the ecosystem through Delta Kernel

External access to UC managed tables is part of a broader investment in the Delta Lake and Unity Catalog OSS ecosystem. The goal is to align Delta with the catalog-managed model pioneered by Iceberg so enterprises can benefit from the performance and consistency guarantees of a catalog while maintaining interoperability with any engine.

We’re expanding the Delta ecosystem with Delta Kernel, which provides libraries for reading, writing, and committing to Delta tables without reimplementing the protocol from scratch. Paired with the UC Delta APIs in UC OSS, Delta Kernel allows an engine to integrate seamlessly with a Databricks-managed catalog or a self-hosted UC OSS deployment.

For example, DuckDB built its Delta and Unity Catalog OSS extensions on Delta Kernel. As of DuckDB v1.5.1, it can read and write UC managed Delta tables directly. This reflects our commitment to an open, interoperable ecosystem: as the Delta protocol evolves, engines built on the Kernel inherit the latest features automatically, so customers always have access to the latest and greatest capabilities no matter which engine they choose.

“DuckDB users value being able to work directly with open lakehouse data from a lightweight analytical engine. Delta Lake and Unity Catalog integration help bring that experience to governed enterprise tables, including workflows where data written through one engine can be read consistently by another.” —Hannes Mühleisen, CEO, DuckDB Labs

We’re continuing to work with the community to develop more Kernel-based integrations with UC managed Delta tables. For example, we’re taking major strides to bring Flink streaming workloads into UC managed tables with a new Kernel-based Flink connector.

Centralized governance

Centralized governance is critical for multi-engine pipelines. A streaming application may write data into a managed table, Spark may transform it, and Starburst or DuckDB may query it — regardless of the engine or the identity behind the query, access controls must be respected.

Unity Catalog is the single governance layer across every engine. Platform teams control which principals can access tables externally and what they can do through UC privileges. Cross-engine ABAC extends this further, allowing Unity Catalog to filter rows or mask columns on reads from external engines. Fine-grained access controls are centrally enforced for UC managed tables with no custom workarounds required.

Start using UC managed tables today

For teams building on the open lakehouse, UC managed tables are the strongest foundation available. Predictive Optimization automatically tunes data layouts and performs table maintenance, delivering best price-performance with no manual effort. Interoperability is also built in: managed tables are stored in the Delta or Iceberg format and are accessible from any engine, so teams never have to choose between performance and flexibility.

Teams on external tables can upgrade existing tables in place with ALTER TABLE SET MANAGED, so they gain Predictive Optimization and catalog commits while keeping the open, multi-engine access they already rely on.

Our vision is clear: customers should be able to use the best engine for each workload while keeping one copy of data in Unity Catalog with the best price-performance and unified governance.

How to Get Started

External access to UC managed Delta tables is in Public Preview, including support for create, read, and write operations from external clients.

To get started:

1. Enable external data access on your Unity Catalog metastore.
2. Grant EXTERNAL USE SCHEMA on the schema containing the UC managed Delta tables you want external engines to access.
3. Connect your engine using Delta 4.3 with the Unity Catalog 0.5.1 client. See the external access documentation.
4. Test creates, reads, and writes to UC managed Delta table with catalog commits enabled from external engines (or convert an existing external table to managed)

Information specific to Delta connectors can be found below: