article Performing DML Operations on Unity Catalog Managed Tables from External Engines in Technical Blog

article Performing DML Operations on Unity Catalog Managed Tables from External Engines in Technical Blog https://community.databricks.com/t5/technical-blog/performing-dml-operations-on-unity-catalog-managed-tables-from/ba-p/156308 <H2><SPAN>Summary</SPAN></H2> <UL> <LI style="font-weight: 400;" aria-level="1"><SPAN>External engines — </SPAN><A href="https://spark.apache.org/" target="_blank" rel="noopener"><SPAN>Apache Spark</SPAN></A><SPAN>™ (batch and Structured Streaming), </SPAN><A href="https://duckdb.org/" target="_blank" rel="noopener"><SPAN>DuckDB</SPAN></A><SPAN>, </SPAN><A href="https://flink.apache.org/" target="_blank" rel="noopener"><SPAN>Apache Flink</SPAN></A><SPAN>, </SPAN><A href="https://docs.starburst.io/starburst-galaxy/working-with-data/create-catalogs/non-object-storage/unity.html" target="_blank" rel="noopener"><SPAN>Starburst</SPAN></A><SPAN>, and </SPAN><A href="https://streamnative.io/" target="_blank" rel="noopener"><SPAN>StreamNative</SPAN></A><SPAN> Kafka Service — can now create, read, and write Unity Catalog-managed Delta tables from outside Databricks, with every commit coordinated by Unity Catalog.</SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><SPAN>UC serializes commits across engines, so concurrent external writers don't corrupt the transaction log; every operation is auditable; and </SPAN><A href="https://docs.databricks.com/aws/en/optimizations/predictive-optimization" target="_blank" rel="noopener"><SPAN>Predictive Optimization</SPAN></A><SPAN> continues to manage table health (compaction, vacuuming, statistics)</SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><SPAN>Ships with </SPAN><A href="https://github.com/delta-io/delta/releases/tag/v4.2.0" target="_blank" rel="noopener"><SPAN>Delta Lake 4.2.0</SPAN></A><SPAN> and </SPAN><A href="https://github.com/unitycatalog/unitycatalog/releases/tag/v0.4.1" target="_blank" rel="noopener"><SPAN>Unity Catalog 0.4.1</SPAN></A><SPAN>. Grant</SPAN><FONT face="courier new,courier"><SPAN> EXTERNAL_USE_SCHEMA</SPAN></FONT><SPAN>, authenticate with M2M OAuth, and external clients are up and running in minutes — reading from and writing to </SPAN><A href="https://docs.databricks.com/aws/en/external-access/unity-rest" target="_blank" rel="noopener"><SPAN>Unity Catalog managed Delta tables</SPAN></A><SPAN> through the Unity REST API.</SPAN></LI> </UL> <P><SPAN>The </SPAN><A href="https://www.databricks.com/product/data-lakehouse" target="_blank" rel="noopener"><SPAN>open data lakehouse</SPAN></A><SPAN> is built on a simple principle: maintain a single copy of your data, govern it centrally, and access it from the engine best suited to each workload. Previously, we launched </SPAN><A href="https://www.databricks.com/blog/secure-external-access-unity-catalog-assets-open-apis" target="_blank" rel="noopener"><SPAN>UC Open APIs</SPAN></A><SPAN> and credential vending, enabling external engines to access tables in Unity Catalog. Now, we are evolving this vision to safely create or write to them.</SPAN></P> <P><SPAN>The Beta of External Access to Unity Catalog Managed Tables with </SPAN><A href="https://docs.databricks.com/aws/en/delta/catalog-commits" target="_blank" rel="noopener"><SPAN>Catalog Commits</SPAN></A><SPAN> extends Unity Catalog’s open APIs to deliver full read and write interoperability for managed Delta tables from external engines. External engines can now create, read, and write to these tables, while Unity Catalog coordinates every transaction to ensure consistency, governance, and safe multi-engine operation.</SPAN></P> <P><SPAN>Enterprises can bring the best engine to every workload and benefit from price/performance optimizations –without moving or duplicating data. This further establishes Unity Catalog as the industry's most open and interoperable catalog: one copy of data, governed centrally, and accessible from any engine, without requiring separate copies for different platforms or workloads.</SPAN></P> <P><SPAN>In this blog, we walk through how to perform DML operations on Unity Catalog managed Delta tables from external engines using Apache Spark™ and DuckDB. You can find the complete code examples in </SPAN><A href="https://github.com/databricks-solutions/databricks-blogposts/tree/main/2026-05-external-access-to-unity-catalog-managed-delta-tables" target="_blank" rel="noopener"><SPAN>this GitHub repository</SPAN></A><SPAN>.</SPAN></P> <P><SPAN>Scope note. This Beta is for managed Delta tables only. Managed Iceberg tables already support external writes through Iceberg REST in Public Preview, so both open formats under Unity Catalog now offer full external read, write, and create from outside Databricks.</SPAN></P> <P><SPAN>Scope of this walkthrough. The Beta currently ships connector integrations for five engines: Apache Spark™, DuckDB, Apache Flink, Starburst, and StreamNative Kafka Service. The hands-on examples below cover Apache Spark™, DuckDB, and Apache Flink end to end. Starburst and StreamNative Kafka Service use the same Delta Kernel + catalog-commits substrate, so the integration is identical; see their connector docs for SDK-specific setup.</SPAN></P> <H2><STRONG>How Unity Catalog Enables Safe External Writes</STRONG></H2> <H3><STRONG>The Challenge of Coordinating Writes</STRONG></H3> <P><SPAN>Delta tables track every change in an ordered transaction log. When two writers append concurrently without a coordinator, they can pick the same log version, corrupt the log, or silently overwrite each other. Reads alone are safe—they're snapshot-isolated by design—but writes from multiple external engines on the same table were not.</SPAN></P> <H3><STRONG>Catalog Commits: A Centralized Commit Coordinator</STRONG></H3> <P><SPAN>Catalog commits solve this by making Unity Catalog the commit coordinator. Every write—regardless of which engine initiated it—flows through Unity Catalog, which serializes commits, enforces access controls, and records each operation for audit. Because Unity Catalog sees every external operation, </SPAN><A href="https://docs.databricks.com/aws/en/optimizations/predictive-optimization" target="_blank" rel="noopener"><SPAN>Predictive Optimization</SPAN></A><SPAN> continues to maintain table health even when tables are written entirely by engines outside Databricks. Catalog commits also lay the foundation for capabilities that require centralized coordination, such as </SPAN><A href="https://docs.databricks.com/aws/en/transactions/" target="_blank" rel="noopener"><SPAN>multi-table transactions</SPAN></A><SPAN>.</SPAN></P> <H3><STRONG>Delta Kernel: The Foundation for Open Connectivity</STRONG></H3> <P><SPAN>The open source library that enables connector authors to make this possible is Delta Kernel—a set of Java and Rust APIs for reading, writing, and committing to Delta tables. Delta Kernel abstracts the low-level Delta protocol, so new connectors need only integrate it into their engine's I/O layer. Apache Spark™ integration (</SPAN><SPAN>unitycatalog-spark</SPAN><SPAN>), Apache Flink, StreamNative, and the DuckDB integration (</SPAN><SPAN>unity_catalog</SPAN><SPAN> core extension) are built on Delta Kernel.</SPAN></P> <H3><SPAN>Authentication: M2M OAuth</SPAN></H3> <P><SPAN>External clients authenticate to Databricks using </SPAN><A href="https://docs.databricks.com/aws/en/dev-tools/auth/oauth-m2m" target="_blank" rel="noopener"><SPAN>machine-to-machine (M2M) OAuth</SPAN></A><SPAN> with a service principal. </SPAN><A href="https://docs.databricks.com/aws/en/dev-tools/auth/pat" target="_blank" rel="noopener"><SPAN>Personal Access Tokens</SPAN></A><SPAN> (PATs) are also supported, but they're legacy, per-user, long-lived, and difficult to rotate. </SPAN></P> <P><SPAN>The Databricks SDK fetches a short-lived access token from the Service Principal's client credentials:</SPAN></P> <LI-CODE lang="python">from databricks.sdk.core import Config, oauth_service_principal def get_oauth_token() -> str: config = Config( host=DATABRICKS_HOST, client_id=DATABRICKS_CLIENT_ID, client_secret=DATABRICKS_CLIENT_SECRET, ) token_source = oauth_service_principal(config) return token_source().access_token</LI-CODE> <P><SPAN>For the Spark connector, you don't need to mint a token yourself — the connector accepts the SP's client credentials and refreshes its own tokens against </SPAN><FONT face="courier new,courier"><SPAN>/oidc/v1/token</SPAN></FONT><SPAN> whenever they expire. This is the production path for long-running pipelines (especially Structured Streaming).</SPAN></P> <H3><SPAN>Setting up the demo catalog</SPAN></H3> <P><SPAN>From a Databricks workspace, create a catalog and schema. For this blog, we have cloned eight </SPAN><SPAN>samples.tpch</SPAN><SPAN> tables as UC-managed Delta tables into the schema, and granted </SPAN><FONT face="courier new,courier"><SPAN>EXTERNAL_USE_SCHEMA</SPAN></FONT><SPAN> on the schema to the Service Principal that the external engines will authenticate as. Full DDL is in </SPAN><FONT face="courier new,courier"><SPAN>00_setup_databricks.sql</SPAN></FONT><SPAN>; the essentials:</SPAN></P> <LI-CODE lang="python">DROP CATALOG IF EXISTS uc_ext_access_demo CASCADE; CREATE CATALOG uc_ext_access_demo; CREATE SCHEMA uc_ext_access_demo.tpch_managed; USE CATALOG uc_ext_access_demo; USE SCHEMA tpch_managed; CREATE OR REPLACE TABLE customer TBLPROPERTIES ('delta.feature.catalogManaged' = 'supported') AS SELECT * FROM samples.tpch.customer; CREATE OR REPLACE TABLE orders TBLPROPERTIES ('delta.feature.catalogManaged' = 'supported') AS SELECT * FROM samples.tpch.orders; -- repeat for lineitem, nation, part, partsupp, region, supplier GRANT USE CATALOG ON CATALOG uc_ext_access_demo TO `<SP_CLIENT_ID>`; GRANT USE SCHEMA, SELECT, MODIFY, CREATE TABLE, EXTERNAL_USE_SCHEMA ON SCHEMA uc_ext_access_demo.tpch_managed TO `<SP_CLIENT_ID>`;</LI-CODE> <P><SPAN>The </SPAN><FONT face="courier new,courier"><SPAN>DROP CATALOG IF EXISTS … CASCADE</SPAN></FONT><SPAN> at the start makes this idempotent — every run wipes prior state and reseeds, so the demo always starts from a known baseline.</SPAN></P> <H2><SPAN>DuckDB — READ and INSERT</SPAN></H2> <P><SPAN>DuckDB connects to Unity Catalog via two core extensions — </SPAN><SPAN>unity_catalog</SPAN><SPAN> (Databricks UC integration) and </SPAN><SPAN>delta</SPAN><SPAN> (Kernel-based Delta reader/writer). Both are installed and loaded at runtime, then a single anonymous </SPAN><SPAN>UNITY_CATALOG</SPAN><SPAN> secret carries the OAuth token, and the </SPAN><SPAN>ATTACH</SPAN><SPAN> integrates the catalog into DuckDB's namespace:</SPAN></P> <LI-CODE lang="python">con = duckdb.connect() con.execute("INSTALL unity_catalog; LOAD unity_catalog;") con.execute("INSTALL delta; LOAD delta;") con.execute(f""" CREATE SECRET ( TYPE UNITY_CATALOG, TOKEN '{get_oauth_token()}', ENDPOINT '{DATABRICKS_HOST}/', AWS_REGION '{AWS_REGION}' ); """) con.execute(""" ATTACH 'uc_ext_access_demo' AS uc_ext_access_demo (TYPE UNITY_CATALOG); """)</LI-CODE> <P><SPAN>Two details that the </SPAN><A href="https://duckdb.org/docs/current/core_extensions/unity_catalog" target="_blank" rel="noopener"><SPAN>official `unity_catalog` extension docs</SPAN></A><SPAN> emphasize: the </SPAN><FONT face="courier new,courier"><SPAN>ENDPOINT</SPAN></FONT><SPAN> is the workspace root URL (not </SPAN><FONT face="courier new,courier"><SPAN><host>/api/2.1/unity-catalog</SPAN></FONT><SPAN>), and the </SPAN><FONT face="courier new,courier"><SPAN>SECRET</SPAN></FONT><SPAN> is created without a name, so the extension auto-resolves it by type when </SPAN><SPAN>ATTACH</SPAN><SPAN> runs without an explicit </SPAN><FONT face="courier new,courier"><SPAN>SECRET</SPAN></FONT><SPAN> reference.</SPAN></P> <P><SPAN>The cloned TPCH tables behave like native DuckDB tables. Read, scan, JOIN — all the standard DuckDB SQL surface works against managed Delta tables:</SPAN></P> <LI-CODE lang="python">con.execute(""" SELECT n.n_name, count(*) AS customer_count FROM uc_ext_access_demo.tpch_managed.customer c JOIN uc_ext_access_demo.tpch_managed.nation n ON c.c_nationkey = n.n_nationkey GROUP BY n.n_name ORDER BY customer_count DESC LIMIT 5 """).fetchdf()</LI-CODE> <P><SPAN>And with </SPAN><FONT face="courier new,courier"><SPAN>INSERT INTO</SPAN></FONT><SPAN> support, DuckDB can now append rows directly to existing Unity Catalog managed Delta tables. UC serializes the commit, the Delta extension runs the actual write through the kernel, and Spark / Databricks readers see the new rows immediately:</SPAN></P> <P><FONT face="courier new,courier"><SPAN>orders = "uc_ext_access_demo.tpch_managed.orders"</SPAN></FONT></P> <LI-CODE lang="python"># Single-row VALUES — three rows in one statement con.execute(f""" INSERT INTO {orders} ( o_orderkey, o_custkey, o_orderstatus, o_totalprice, o_orderdate, o_orderpriority, o_clerk, o_shippriority, o_comment ) VALUES (9000000301, 1, 'O', 100.00, DATE '2026-04-23', '3-MEDIUM', 'Clerk#external-duckdb', 0, 'duckdb single-row insert'), (9000000302, 2, 'O', 200.00, DATE '2026-04-23', '3-MEDIUM', 'Clerk#external-duckdb', 0, 'duckdb single-row insert'), (9000000303, 3, 'O', 300.00, DATE '2026-04-23', '5-LOW', 'Clerk#external-duckdb', 0, 'duckdb single-row insert'); """) # Bulk insert from another UC table con.execute(f""" INSERT INTO {orders} SELECT o_orderkey + 9_000_000_400 AS o_orderkey, o_custkey, o_orderstatus, o_totalprice, o_orderdate, o_orderpriority, 'Clerk#external-duckdb-bulk' AS o_clerk, o_shippriority, 'duckdb bulk insert' AS o_comment FROM {orders} WHERE o_orderkey BETWEEN 11396166 AND 11396175 """)</LI-CODE> <P><SPAN>The DuckDB </SPAN><FONT face="courier new,courier"><SPAN>unity_catalog</SPAN></FONT><SPAN> extension is currently focused on the read and <FONT face="courier new,courier">INSERT</FONT> path against Databricks UC; the </SPAN><A href="https://duckdb.org/docs/current/core_extensions/unity_catalog" target="_blank" rel="noopener"><SPAN>extension docs</SPAN></A><SPAN> are the source of truth for what's supported as the integration evolves.</SPAN></P> <H2><SPAN>Apache Spark™ — batch read</SPAN></H2> <P><SPAN>The Spark connector for Unity Catalog (</SPAN><SPAN>unitycatalog-spark</SPAN><SPAN>) is the reference implementation for catalog commits. Configure a Spark session outside Databricks, point it at your workspace, and let the connector handle the OAuth flow:</SPAN></P> <LI-CODE lang="python">spark = ( SparkSession.builder.appName("uc-external-access-demo") .config("spark.jars.packages", "io.delta:delta-spark_2.13:4.2.0," "io.unitycatalog:unitycatalog-spark_2.13:0.4.1," "org.apache.hadoop:hadoop-aws:3.4.2") .config("spark.sql.extensions", "io.delta.sql.DeltaSparkSessionExtension") .config("spark.sql.catalog.spark_catalog", "org.apache.spark.sql.delta.catalog.DeltaCatalog") # Register UC as a Spark catalog with M2M OAuth .config(f"spark.sql.catalog.{UC_CATALOG}", "io.unitycatalog.spark.UCSingleCatalog") .config(f"spark.sql.catalog.{UC_CATALOG}.uri", DATABRICKS_HOST) .config(f"spark.sql.catalog.{UC_CATALOG}.auth.type", "oauth") .config(f"spark.sql.catalog.{UC_CATALOG}.auth.oauth.uri", f"{DATABRICKS_HOST}/oidc/v1/token") .config(f"spark.sql.catalog.{UC_CATALOG}.auth.oauth.clientId", DATABRICKS_CLIENT_ID) .config(f"spark.sql.catalog.{UC_CATALOG}.auth.oauth.clientSecret", DATABRICKS_CLIENT_SECRET) .config(f"spark.sql.catalog.{UC_CATALOG}.renewCredential.enabled", "true") .config("spark.sql.defaultCatalog", UC_CATALOG) # UC returns s3:// URIs; map them to Hadoop's S3A FileSystem .config("spark.hadoop.fs.s3.impl", "org.apache.hadoop.fs.s3a.S3AFileSystem") .config("spark.hadoop.fs.AbstractFileSystem.s3.impl", "org.apache.hadoop.fs.s3a.S3A") ).getOrCreate() </LI-CODE> <P><SPAN>A few config lines worth calling out: </SPAN><FONT face="courier new,courier"><SPAN>auth.type=oauth</SPAN></FONT><SPAN> is what tells the connector to mint and refresh tokens itself, so a long-running session doesn't expire mid-job. And </SPAN><FONT face="courier new,courier"><SPAN>spark.hadoop.fs.s3.impl=S3AFileSystem</SPAN></FONT><SPAN> is the bridge between UC's </SPAN><FONT face="courier new,courier"><SPAN>s3://</SPAN></FONT><SPAN> URIs and Hadoop's S3A driver.</SPAN></P> <P><SPAN>Once that's established, reading a managed Delta table from the external Spark process looks identical to reading any other Spark table — every operation goes through UC behind the scenes:</SPAN></P> <LI-CODE lang="python">spark.table("uc_ext_access_demo.tpch_managed.orders").show(5) spark.sql( "DESCRIBE HISTORY uc_ext_access_demo.tpch_managed.orders" ).show(truncate=False)</LI-CODE> <P><SPAN>The </SPAN><FONT face="courier new,courier"><SPAN>DESCRIBE HISTORY</SPAN></FONT><SPAN> output will show </SPAN><FONT face="courier new,courier"><SPAN>engineInfo = Apache-Spark/4.1.1 Delta-Lake/4.2.0</SPAN></FONT><SPAN> for any commit produced by this external session — alongside the original </SPAN><FONT face="courier new,courier"><SPAN>Databricks-Runtime/...</SPAN></FONT><SPAN> commit that seeded the table using a </SPAN><A href="https://docs.databricks.com/aws/en/compute/sql-warehouse/" target="_blank" rel="noopener"><SPAN>Databricks SQL Warehouse</SPAN></A><SPAN>.</SPAN></P> <H2><SPAN>Apache Spark™ — batch write</SPAN></H2> <P><SPAN>The same external session can append rows or create a brand-new managed Delta table with </SPAN><FONT face="courier new,courier"><SPAN>CREATE TABLE AS SELECT</SPAN></FONT><SPAN>:</SPAN></P> <LI-CODE lang="python">new_df.write.mode("append").saveAsTable( "uc_ext_access_demo.tpch_managed.orders" ) spark.sql(""" CREATE TABLE uc_ext_access_demo.tpch_managed.orders_summary USING DELTA TBLPROPERTIES ('delta.feature.catalogManaged' = 'supported') AS SELECT o_orderstatus, o_orderpriority, count(*) AS order_count, sum(o_totalprice) AS total_value, avg(o_totalprice) AS avg_value FROM uc_ext_access_demo.tpch_managed.orders GROUP BY o_orderstatus, o_orderpriority """)</LI-CODE> <P><SPAN>Two things worth noting:</SPAN></P> <UL> <LI style="font-weight: 400;" aria-level="1"><SPAN>`USING DELTA TBLPROPERTIES ('delta.feature.catalogManaged' = 'supported')` is required when an external engine creates a managed Delta table for the first time. This is the table-level feature flag that enables catalog-managed commits — UC would otherwise reject the create managed table request.</SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><SPAN>`saveAsTable` DataFrame writer just works for appending to existing managed tables; no DDL feature flag needed at append time because the property is already set.</SPAN></LI> </UL> <P><SPAN>Once both writes are complete, </SPAN><FONT face="courier new,courier"><SPAN>DESCRIBE HISTORY</SPAN></FONT><SPAN> shows the original Databricks-Runtime CTAS sitting alongside the new </SPAN><FONT face="courier new,courier"><SPAN>Apache-Spark/4.1.1 Delta-Lake/4.2.0</SPAN></FONT><SPAN> commits — same managed table, multiple engine </SPAN><SPAN>writes with catalog commits.</SPAN></P> <H2><SPAN>Apache Spark™ — Structured Streaming</SPAN></H2> <P><SPAN>Catalog-managed commits also cover Structured Streaming writes. Each micro-batch becomes a single UC commit, attributed to the external Spark engine. The pattern below uses Spark's built-in </SPAN><SPAN>rate</SPAN><SPAN> source, so the example is self-contained — no external broker required — but the exact same code shape works for Kafka, Kinesis, files, anything Spark can stream from.</SPAN></P> <LI-CODE lang="python">spark.sql(""" CREATE TABLE uc_ext_access_demo.tpch_managed.orders_stream ( o_orderkey BIGINT, o_custkey BIGINT, o_orderstatus STRING, o_totalprice DECIMAL(18,2), o_orderdate DATE, o_orderpriority STRING, o_clerk STRING, o_shippriority INT, o_comment STRING ) USING DELTA TBLPROPERTIES ('delta.feature.catalogManaged' = 'supported') """) src=( spark.readStream.format("rate") .option("rowsPerSecond", 50) .load() ) shaped = ( src .withColumn("o_orderkey", (lit(9_100_000_000) + col("value")).cast("bigint")) .withColumn("o_custkey", ((col("value") % lit(1500)) + lit(1)).cast("bigint")) .withColumn("o_orderstatus", lit("O")) .withColumn("o_totalprice", expr("cast(rand() * 1000 as decimal(18,2))")) .withColumn("o_orderdate", current_date()) .withColumn("o_orderpriority", lit("3-MEDIUM")) .withColumn("o_clerk", lit("Clerk#external-spark-stream")) .withColumn("o_shippriority", lit(0)) .withColumn("o_comment", concat(lit("streamed by external Spark @ batch_value="), col("value").cast("string"))) .select("o_orderkey", "o_custkey", "o_orderstatus", "o_totalprice", "o_orderdate", "o_orderpriority", "o_clerk", "o_shippriority", "o_comment") ) query = ( shaped.writeStream .format("delta") .outputMode("append") .option("checkpointLocation", "/tmp/orders_stream_ckpt") .trigger(processingTime="5 seconds") .toTable("uc_ext_access_demo.tpch_managed.orders_stream") ) query.awaitTermination(30) # run for ~30 seconds, then exit</LI-CODE> <P><SPAN>After the stream stops, </SPAN><FONT face="courier new,courier"><SPAN>DESCRIBE HISTORY</SPAN></FONT><SPAN> on </SPAN><FONT face="courier new,courier"><SPAN>orders_stream</SPAN></FONT><SPAN> shows the </SPAN><FONT face="courier new,courier"><SPAN>CREATE TABLE</SPAN></FONT><SPAN> commit (version 0) followed by a </SPAN><FONT face="courier new,courier"><SPAN>STREAMING UPDATE</SPAN></FONT><SPAN> for each micro-batch — everyone tagged </SPAN><FONT face="courier new,courier"><SPAN>Apache-Spark/4.1.1 Delta-Lake/4.2.0</SPAN></FONT><SPAN>. The checkpoint is a local directory; UC only coordinates the catalog commit itself, not the engine's micro-batch state.</SPAN></P> <H2><SPAN>Apache Flink — Streaming Sink to a UC Managed Delta Table</SPAN></H2> <P><SPAN>Delta Lake 4.2 ships a brand-new Flink connector built on Delta Kernel — the legacy Flink connector deprecated in Delta 4.0 alongside Delta Standalone is gone, and the new connector supports Unity Catalog managed Delta tables from inception. It is sink-only and experimental today, with exactly-once semantics backed by a Flink Sink Writer and a single global Committer — every Flink checkpoint becomes one UC commit on the target table. The connector accepts a bearer credential through the Flink catalog's token field. Personal Access Tokens (PATs) work here, and so do machine-to-machine (M2M) OAuth bearer tokens — this demo passes the M2M OAuth bearer token used by the DuckDB and Spark sections (PATs are considered legacy at Databricks).</SPAN></P> <P><SPAN>The pattern below pairs a bounded datagen source (so the job terminates cleanly without an explicit STOP JOB) with a UC-managed Delta sink created up front with delta.feature.catalogManaged='supported'. Each Flink checkpoint produces a real Delta version coordinated by UC.</SPAN></P> <LI-CODE lang="python">SET 'table.dml-sync' = 'true'; -- Register Unity Catalog as a Flink catalog. The Flink catalog name must -- match the UC catalog name so 3-part identifiers resolve straight through. CREATE CATALOG `uc_ext_access_demo` WITH ( 'type' = 'unitycatalog', 'endpoint' = '<DATABRICKS_HOST>/', 'token' = '<BEARER_TOKEN>' ); CREATE TEMPORARY TABLE flink_orders_flink_src ( seq BIGINT, payload STRING ) WITH ( 'connector' = 'datagen', 'rows-per-second' = '100', 'fields.seq.kind' = 'sequence', 'fields.seq.start' = '1', 'fields.seq.end' = '3000', 'fields.payload.length' = '24' ); INSERT INTO `uc_ext_access_demo`.`tpch_managed`.`orders_flink` SELECT CAST(9200000000 + seq AS BIGINT) AS o_orderkey, CAST((seq % 1500) + 1 AS BIGINT) AS o_custkey, CAST('O' AS STRING) AS o_orderstatus, CAST(MOD(seq * 7, 1000) AS DECIMAL(18,2)) AS o_totalprice, CURRENT_DATE AS o_orderdate, CAST('3-MEDIUM' AS STRING) AS o_orderpriority, CAST('Clerk#external-flink-stream' AS STRING) AS o_clerk, CAST(0 AS INT) AS o_shippriority, CONCAT('streamed by external Flink @ seq=', CAST(seq AS STRING)) AS o_comment FROM flink_orders_flink_src; -- --------------------------------------------------------------------------- -- Demo 2 — small INSERT into the shared `orders` table (also written by -- Spark + DuckDB). After this, DESCRIBE HISTORY orders shows -- commits from FOUR engines — Databricks Runtime, external -- Apache Spark, DuckDB, and Apache Flink — all coordinated by UC. -- --------------------------------------------------------------------------- CREATE TEMPORARY TABLE flink_orders_shared_src ( seq BIGINT ) WITH ( 'connector' = 'datagen', 'rows-per-second' = '10', 'fields.seq.kind' = 'sequence', 'fields.seq.start' = '1', 'fields.seq.end' = '10' ); INSERT INTO `uc_ext_access_demo`.`tpch_managed`.`orders` SELECT CAST(9300000000 + seq AS BIGINT) AS o_orderkey, CAST((seq % 1500) + 1 AS BIGINT) AS o_custkey, CAST('O' AS STRING) AS o_orderstatus, CAST(MOD(seq * 11, 1000) AS DECIMAL(18,2)) AS o_totalprice, CURRENT_DATE AS o_orderdate, CAST('3-MEDIUM' AS STRING) AS o_orderpriority, CAST('Clerk#external-flink-stream' AS STRING) AS o_clerk, CAST(0 AS INT) AS o_shippriority, CONCAT('inserted into shared orders by external Flink @ seq=', CAST(seq AS STRING)) AS o_comment FROM flink_orders_shared_src;</LI-CODE> <P><SPAN>The connector ships on Maven Central as <FONT face="courier new,courier">io.delta:delta-flink</FONT> (4.2.0 at the time of writing — <FONT face="courier new,courier"><A href="https://central.sonatype.com/artifact/io.delta/delta-flink" target="_blank" rel="noopener">https://central.sonatype.com/artifact/io.delta/delta-flink</A></FONT>). It is a thin JAR — its transitive deps (<FONT face="courier new,courier">delta-kernel-*, hadoop-aws, AWS SDK bundle, parquet, ...</FONT>) must also be on the Flink classpath. The companion helper setup_flink_usrlib.sh wraps mvn dependency:copy-dependencies and stages the full set under <FONT face="courier new,courier">flink/docker/<ver>/usrlib/</FONT> in one shot; building from source via sbt flink/assembly is documented as an alternative for master features. Setting '<FONT face="courier new,courier">table.dml-sync</FONT>' to '<FONT face="courier new,courier">true</FONT>' causes the Flink SQL client to block on the streaming <FONT face="courier new,courier">INSERT</FONT>, so the run terminates cleanly when the bounded datagen source completes.</SPAN></P> <P><SPAN>After the run, <FONT face="courier new,courier">DESCRIBE HISTORY </FONT></SPAN><FONT face="courier new,courier"><SPAN>uc_ext_access_demo.tpch_managed.orders_flink</SPAN></FONT><SPAN> shows version 0 (the <FONT face="courier new,courier">CREATE TABLE</FONT>) followed by the Flink-attributed commit(s) — proof that Unity Catalog coordinated the external Flink write through catalog commits.</SPAN></P> <H2><SPAN>Auditing Cross-Engine Writes with </SPAN><SPAN>DESCRIBE HISTORY</SPAN></H2> <P><SPAN>The end-to-end demo seeds </SPAN><SPAN>orders</SPAN><SPAN> with a Databricks-Runtime CTAS, then has DuckDB, external Apache Spark™, and Apache Flink take turns writing to it. After that, a single </SPAN><FONT face="courier new,courier"><SPAN>DESCRIBE HISTORY orders</SPAN></FONT><SPAN> from any of them shows the full history of operations log — the </SPAN><FONT face="courier new,courier"><SPAN>engineInfo</SPAN></FONT><SPAN> column tells exactly which engine produced each commit.</SPAN></P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="dkushari_0-1778088020110.png" style="width: 999px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/26711i9C5092CCBEF7A4D1/image-size/large?v=v2&px=999" role="button" title="dkushari_0-1778088020110.png" alt="dkushari_0-1778088020110.png" /></span></P> <P><SPAN>Multiple commits, four engines, one log. Unity Catalog was the coordinator for every entry — that's the cross-engine guarantee. One small note on the table above:</SPAN></P> <UL> <LI style="font-weight: 400;" aria-level="1"><SPAN>The </SPAN><FONT face="courier new,courier"><SPAN>Databricks-Runtime/...</SPAN></FONT><SPAN> row at version 0 came from a serverless DBSQL warehouse running the setup SQL. Anything written inside Databricks identifies itself with that Databricks runtime string in </SPAN><SPAN>engineInfo</SPAN><SPAN>; anything written from external engine is identified with the engine + connector versions (here, <FONT face="courier new,courier">Apache-Spark/4.1.1 Delta-Lake/4.2.0, DuckDB, and Kernel-<ver>/DeltaSink for Flink</FONT>).</SPAN></LI> </UL> <P><SPAN>The queries below tag every row in the </SPAN><SPAN>orders</SPAN><SPAN> table with the engine that wrote it, using the </SPAN><FONT face="courier new,courier"><SPAN>o_clerk</SPAN></FONT><SPAN> column to identify each writer engine. The first query returns a per-engine count, and the second one returns a small sample of rows to show the actual records inserted by each engine.</SPAN></P> <P><U>Query 1:</U></P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="dkushari_1-1778088020110.png" style="width: 999px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/26712i7774C77CB81171A3/image-size/large?v=v2&px=999" role="button" title="dkushari_1-1778088020110.png" alt="dkushari_1-1778088020110.png" /></span></P> <P><U>Query 2:</U></P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="dkushari_4-1778089724668.png" style="width: 999px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/26722i12F90868A1A0C73D/image-size/large?v=v2&px=999" role="button" title="dkushari_4-1778089724668.png" alt="dkushari_4-1778089724668.png" /></span><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="dkushari_2-1778088020110.png" style="width: 999px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/26713iD957BB19FB6F1202/image-size/large?v=v2&px=999" role="button" title="dkushari_2-1778088020110.png" alt="dkushari_2-1778088020110.png" /></span></P> <H2><SPAN>Ecosystem status and direction</SPAN></H2> <P><SPAN>The ecosystem continues to evolve:</SPAN></P> <UL> <LI style="font-weight: 400;" aria-level="1"><STRONG>Delta Kernel remains the common building block.</STRONG><SPAN> New connectors only need to bridge the kernel into their engine’s I/O layer.</SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><STRONG>Apache Flink and StreamNative Kafka Service already include catalog-commit integrations.</STRONG><SPAN> These systems follow the same interoperability pattern, with Delta Kernel handling the core abstraction and Unity Catalog acting as the commit coordinator.</SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><STRONG>Attribute-based access control (ABAC) for external reads is the next governance frontier.</STRONG><SPAN> The goal is to extend Unity Catalog’s row- and column-level policies to queries executed by external engines against managed tables.</SPAN></LI> </UL> <H2><SPAN>Conclusion</SPAN></H2> <P><SPAN>With this Beta, the open lakehouse story is finally symmetric. Store your data once as a UC-managed Delta table, govern it centrally, and let every engine — Apache Spark™ in batch, Spark Structured Streaming, DuckDB, or whatever Delta Kernel-backed engine comes next — perform DML operations against the same tables. Catalog commits make it safe; M2M OAuth and </SPAN><FONT face="courier new,courier"><SPAN>EXTERNAL_USE_SCHEMA</SPAN></FONT><SPAN> make it governed; Delta Kernel makes it easy to extend.</SPAN></P> <P><SPAN>Clone the </SPAN><A href="https://github.com/databricks-solutions/databricks-blogposts/tree/main/2026-05-external-access-to-unity-catalog-managed-delta-tables" target="_blank" rel="noopener"><SPAN>GitHub repository</SPAN></A><SPAN>, and try performing DML operations from Spark or DuckDB today. Note the output of </SPAN><FONT face="courier new,courier"><SPAN>DESCRIBE HISTORY</SPAN></FONT><SPAN> on a table that shows commits from Databricks Runtime, DuckDB, external Spark, and Apache Flink in the same log, demonstrating a single, interleaved transaction history where each commit is attributed to the</SPAN><FONT face="courier new,courier"><SPAN> engineInfo</SPAN></FONT><SPAN> column, proving cross-engine write coordination under Unity Catalog.</SPAN></P> <H2><SPAN>Appendix</SPAN></H2> <H3><SPAN>Code and execution flow</SPAN></H3> <P><SPAN>A full, runnable demo is available in </SPAN><A href="https://github.com/databricks-solutions/databricks-blogposts/tree/main/2026-05-external-access-to-unity-catalog-managed-delta-tables" target="_blank" rel="noopener"><SPAN>this GitHub repository</SPAN></A><SPAN>.</SPAN></P> <P> </P> <TABLE> <TBODY> <TR> <TD> <P><SPAN>File</SPAN></P> </TD> <TD> <P><SPAN>Purpose</SPAN></P> </TD> </TR> <TR> <TD> <P><SPAN>00_setup_databricks.sql</SPAN></P> </TD> <TD> <P><SPAN><FONT face="courier new,courier">DROP</FONT> + <FONT face="courier new,courier">CREATE</FONT> catalog + schema, clone 8 TPCH tables as managed Delta, grant <FONT face="courier new,courier">EXTERNAL_USE_SCHEMA</FONT>.</SPAN></P> </TD> </TR> <TR> <TD> <P><SPAN>run_setup.py</SPAN></P> </TD> <TD> <P><SPAN>Driver: substitutes the SP application_id, splits + submits the SQL via the Statement Execution API.</SPAN></P> </TD> </TR> <TR> <TD> <P><SPAN>01_duckdb_read.py</SPAN></P> </TD> <TD> <P><SPAN>DuckDB — list / <FONT face="courier new,courier">SELECT</FONT> / cross-table <FONT face="courier new,courier">JOIN</FONT> against UC managed Delta tables.</SPAN></P> </TD> </TR> <TR> <TD> <P><SPAN>02_duckdb_insert.py</SPAN></P> </TD> <TD> <P><SPAN>DuckDB — <FONT face="courier new,courier">INSERT INTO ... VALUES</FONT> and <FONT face="courier new,courier">INSERT INTO ... SELECT</FONT> into a managed Delta table.</SPAN></P> </TD> </TR> <TR> <TD> <P><SPAN>03_spark_external_read.py</SPAN></P> </TD> <TD> <P><SPAN>External Apache Spark™ — batch read + <FONT face="courier new,courier">DESCRIBE HISTORY</FONT> + time travel via <FONT face="courier new,courier">VERSION AS OF</FONT>.</SPAN></P> </TD> </TR> <TR> <TD> <P><SPAN>04_spark_external_write.py</SPAN></P> </TD> <TD> <P><SPAN>External Apache Spark™ — append + <FONT face="courier new,courier">CREATE TABLE AS SELECT</FONT>.</SPAN></P> </TD> </TR> <TR> <TD> <P><SPAN>05_spark_streaming.py</SPAN></P> </TD> <TD> <P><SPAN>External Apache Spark™ Structured Streaming — <FONT face="courier new,courier">readStream</FONT> (rate source) → <FONT face="courier new,courier">writeStream.toTable()</FONT> into a managed Delta table; ends with a time-travel comparison.</SPAN></P> </TD> </TR> <TR> <TD> <P><SPAN>06_flink_streaming.py</SPAN></P> </TD> <TD> <P><SPAN>External Apache Flink — drop+create orders_flink, <FONT face="courier new,courier">DELETE</FONT> prior Flink-marker rows from orders, render Flink SQL using the catalog-based form (<FONT face="courier new,courier">CREATE CATALOG ... WITH 'type'='unitycatalog'</FONT> + <FONT face="courier new,courier">INSERT INTO <catalog>.<schema>.<table></FONT>) with the OAuth bearer token substituted, then auto-detect a running Flink JobManager Docker container and exec bin/sql-client.sh -f against it. The new Kernel-based Delta Flink Connector ships on Maven Central as <FONT face="courier new,courier">io.delta:delta-flink:4.2.0</FONT> — sink-only, experimental.</SPAN></P> </TD> </TR> <TR> <TD> <P><SPAN>07_verify_cross_engine.py</SPAN></P> </TD> <TD> <P><SPAN>External Spark — cross-engine <FONT face="courier new,courier">DESCRIBE HISTORY</FONT> showing the mix of engineInfo values across the catalog.</SPAN></P> </TD> </TR> <TR> <TD> <P><SPAN>run_all.py</SPAN></P> </TD> <TD> <P><SPAN>One-shot pipeline runner — 9 steps: setup → duckdb read → duckdb insert → spark read → spark write → streaming → flink → verify → cleanup. Idempotent end-to-end. CLI flags: -a (all), -s (spark), -f (flink), -d (duckdb), --no-setup, --no-cleanup; combinable. At least one engine flag is required (no implicit run-all).</SPAN></P> </TD> </TR> <TR> <TD> <P><SPAN>99_cleanup.sql</SPAN></P> </TD> <TD> <P><SPAN><FONT face="courier new,courier">DROP SCHEMA + CATALOG IF EXISTS … CASCADE</FONT>. Driven by run_cleanup.py at the end of run_all.py, or runnable standalone via the Databricks SQL editor.</SPAN></P> </TD> </TR> <TR> <TD> <P><SPAN>run_cleanup.py</SPAN></P> </TD> <TD> <P><SPAN>Driver: <FONT face="courier new,courier">executes 99_cleanup.sql</FONT> via the Statement Execution API. Runs as the final step of <FONT face="courier new,courier">run_all.py</FONT>, so an end-to-end run leaves the workspace in its pre-demo state.</SPAN></P> </TD> </TR> <TR> <TD> <P><SPAN>setup_flink_usrlib.sh</SPAN></P> </TD> <TD> <P><SPAN>One-shot Maven helper — runs mvn dependency:copy-dependencies against <FONT face="courier new,courier">io.delta:delta-flink:4.2.0</FONT> and stages the connector + transitive deps (delta-kernel-*, hadoop-aws, AWS SDK bundle, parquet, ...) into a target Flink <FONT face="courier new,courier">usrlib/</FONT> directory. Replaces the manual sbt flink/assembly path for blog readers.</SPAN></P> </TD> </TR> <TR> <TD> <P><SPAN>_common.py</SPAN></P> </TD> <TD> <P><SPAN>Shared OAuth, Spark builder, DuckDB attach helpers.</SPAN></P> </TD> </TR> <TR> <TD> <P><SPAN>requirements.txt</SPAN></P> </TD> <TD> <P><SPAN>Pinned Python package versions.</SPAN></P> </TD> </TR> <TR> <TD> <P><SPAN>.env.example</SPAN></P> </TD> <TD> <P><SPAN>Template for workspace host + service principal credentials.</SPAN></P> </TD> </TR> <TR> <TD> <P><SPAN>README.md</SPAN></P> </TD> <TD> <P><SPAN>End-to-end run instructions.</SPAN></P> </TD> </TR> </TBODY> </TABLE> <H3><SPAN>Prerequisites to run this demo</SPAN></H3> <P><SPAN>Here are the prerequisites to run this demo in your own workspace:</SPAN></P> <UL> <LI style="font-weight: 400;" aria-level="1"><SPAN>A Databricks workspace with Unity Catalog enabled.</SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><SPAN>Workspace enrolled in the </SPAN><SPAN>External Access to Unity Catalog Managed Delta Table</SPAN><SPAN> preview (Settings → Previews).</SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><I><SPAN>External Data Access</SPAN></I><SPAN> enabled on the metastore (Governance → Metastore details).</SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><SPAN>A service principal with an OAuth client_id + client_secret (Workspace admin → Identity and access → Service principals → OAuth secrets).</SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><SPAN><FONT face="courier new,courier">EXTERNAL_USE_SCHEMA, MODIFY</FONT>, </SPAN><SPAN>and</SPAN><FONT face="courier new,courier"><SPAN> SELECT</SPAN></FONT><SPAN> permissions are granted to the </SPAN><SPAN>principal accessing data externally</SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><SPAN>A serverless SQL warehouse (used by the setup script to run the catalog/schema/grants DDL).</SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><SPAN>Local machine with Python 3.11+ and Java 17+.</SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><SPAN>Apache Flink demo prerequisites:</SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><SPAN>Java 17+, Maven (brew install maven), and Docker Desktop installed locally.</SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><SPAN>Stage the Delta Flink Connector + transitive deps from Maven Central into your Flink usrlib/ using the companion helper <FONT face="courier new,courier">scripts/setup_flink_usrlib.sh</FONT> — it runs mvn dependency:copy-dependencies against <FONT face="courier new,courier">io.delta:delta-flink:4.2.0</FONT> (<A href="https://central.sonatype.com/artifact/io.delta/delta-flink" target="_self">https://central.sonatype.com/artifact/io.delta/delta-flink</A>) and copies the full set of JARs (delta-kernel-*, hadoop-aws, AWS SDK bundle, parquet, ...) into the target directory. Building from source via sbt flink/assembly is documented in the upstream README (</SPAN><A href="https://github.com/delta-io/delta/tree/master/flink" target="_blank" rel="noopener"><SPAN>https://github.com/delta-io/delta/tree/master/flink</SPAN></A><SPAN>) as an alternative for master features.</SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><SPAN>Clone delta-io/delta locally to get the docker-compose recipe under flink/docker/<flink-version>/, then run <FONT face="courier new,courier">setup_flink_usrlib.sh</FONT> against that usrlib/ directory.</SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><SPAN>Start the local Flink cluster with docker compose up -d from flink/docker/<flink-version>/. The <FONT face="courier new,courier">06_flink_streaming.py</FONT> script auto-detects the running JobManager container, copies the rendered SQL into it, and runs <FONT face="courier new,courier">bin/sql-client.sh -f</FONT>.</SPAN></LI> </UL> <H3><SPAN>Building and running the Flink demo locally</SPAN></H3> <P><SPAN>The Apache Flink walkthrough above is the only engine in this post that requires a one-time Flink cluster setup beyond the Python venv. The steps below assume you have already completed the workspace setup in Prerequisites for this demo. You will also need Java 17+ and Maven, and Docker installed locally.</SPAN></P> <OL> <LI><SPAN> Install prereqs (one-time). On macOS:</SPAN></LI> </OL> <LI-CODE lang="python">brew install maven</LI-CODE> <P><SPAN># Java 17+ should already be installed; otherwise: </SPAN></P> <LI-CODE lang="python">brew install openjdk@17</LI-CODE> <P><SPAN># Docker Desktop: <A href="https://www.docker.com/products/docker-desktop/" target="_blank" rel="noopener">https://www.docker.com/products/docker-desktop/</A></SPAN></P> <OL start="2"> <LI><SPAN> Clone delta-io/delta locally to get the docker-compose recipe.</SPAN></LI> </OL> <LI-CODE lang="python">git clone https://github.com/delta-io/delta ~/delta</LI-CODE> <OL start="3"> <LI><SPAN> Stage the Delta Flink Connector + transitive deps from Maven Central into the Flink docker-compose usrlib/. The companion helper resolves io.delta:delta-flink:4.2.0 and its transitive deps (delta-kernel-*, hadoop-aws, AWS SDK bundle, parquet, ...) and copies them in one shot.</SPAN></LI> </OL> <LI-CODE lang="python">bash <path-to-blog>/scripts/setup_flink_usrlib.sh ~/delta/flink/docker/2.0/usrlib</LI-CODE> <OL start="4"> <LI><SPAN> Start the local Flink cluster.</SPAN></LI> </OL> <LI-CODE lang="python">cd ~/delta/flink/docker/2.0 docker compose up -d docker compose ps</LI-CODE> <OL start="5"> <LI><SPAN> Run the Flink demo. The script renders the Flink SQL with your live OAuth bearer token, drops+creates orders_flink on Databricks, deletes prior Flink-marker rows from orders, then docker-execs the Flink SQL client against the rendered file.</SPAN></LI> </OL> <LI-CODE lang="markup">python 06_flink_streaming.py</LI-CODE> <OL start="6"> <LI><SPAN> Verify and tear down. The cross-engine verify shows <FONT face="courier new,courier">DESCRIBE HISTORY</FONT> orders with all four engineInfo values, and <FONT face="courier new,courier">DESCRIBE HISTORY</FONT> orders_flink with the Flink-attributed bulk commit.</SPAN></LI> </OL> <LI-CODE lang="python">python 07_verify_cross_engine.py docker compose down # stop the local Flink cluster</LI-CODE> <P><SPAN>For master builds (e.g., testing pre-release Flink connector features), the upstream <FONT face="courier new,courier">sbt flink/assembly</FONT> path still works — see <A href="https://github.com/delta-io/delta/tree/master/flink" target="_self">https://github.com/delta-io/delta/tree/master/flink</A> — but for the blog walkthrough, the Maven Central recipe above is the recommended route.</SPAN></P> <H3><SPAN>References</SPAN></H3> <UL> <LI style="font-weight: 400;" aria-level="1"><A href="https://github.com/delta-io/delta/releases/tag/v4.2.0" target="_blank" rel="noopener"><SPAN>Delta Lake 4.2.0 release notes</SPAN></A></LI> <LI style="font-weight: 400;" aria-level="1"><A href="https://github.com/unitycatalog/unitycatalog/releases/tag/v0.4.1" target="_blank" rel="noopener"><SPAN>Unity Catalog 0.4.1 release notes</SPAN></A></LI> <LI style="font-weight: 400;" aria-level="1"><A href="https://docs.databricks.com/aws/en/external-access/unity-rest" target="_blank" rel="noopener"><SPAN>Databricks External Access documentation</SPAN></A></LI> <LI style="font-weight: 400;" aria-level="1"><A href="https://docs.databricks.com/aws/en/delta/catalog-commits" target="_blank" rel="noopener"><SPAN>Catalog commits docs</SPAN></A></LI> <LI style="font-weight: 400;" aria-level="1"><A href="https://duckdb.org/docs/current/core_extensions/unity_catalog" target="_blank" rel="noopener"><SPAN>DuckDB `unity_catalog` extension docs</SPAN></A></LI> <LI style="font-weight: 400;" aria-level="1"><A href="https://github.com/delta-io/delta/tree/master/flink" target="_blank" rel="noopener"><SPAN>Delta Flink Connector</SPAN></A></LI> <LI style="font-weight: 400;" aria-level="1"><A href="https://streamnative.io/blog/announcing-streamnative-kafka-service-launch-partners#data-platforms-and-lakehouse-ecosystem-partners" target="_blank" rel="noopener"><SPAN>StreamNative Kafka Service Launch Partners</SPAN></A></LI> <LI style="font-weight: 400;" aria-level="1"><A href="https://github.com/delta-io/delta/tree/master/kernel" target="_blank" rel="noopener"><SPAN>Delta Kernel repository</SPAN></A></LI> <LI style="font-weight: 400;" aria-level="1"><A href="https://duckdb.org/2026/05/07/delta-uc-updates" target="_self"><SPAN>DuckDB blog announcing UC integration</SPAN></A></LI> </UL> Fri, 08 May 2026 11:55:14 GMT dkushari 2026-05-08T11:55:14Z Performing DML Operations on Unity Catalog Managed Tables from External Engines https://community.databricks.com/t5/technical-blog/performing-dml-operations-on-unity-catalog-managed-tables-from/ba-p/156308 <P><SPAN>External engines can now create, read, write, and stream into UC-managed Delta tables.</SPAN></P> Fri, 08 May 2026 11:55:14 GMT https://community.databricks.com/t5/technical-blog/performing-dml-operations-on-unity-catalog-managed-tables-from/ba-p/156308 dkushari 2026-05-08T11:55:14Z