[Lakeflow Spark Declarative Pipelines] - Compatibility Mode not working

[Jotaefe1991]

I’m working with an SDP pipeline that creates a streaming table using the dlt.create_streaming_table decorator. My goal is to expose this table through an external location so that a client can read it from Snowflake.

I attempted to configure this directly in the table definition by adding table_properties like this:

 

table_properties={

    "delta.universalFormat.enabledFormats": "compatibility",

    "delta.universalFormat.compatibility.location": f"{compatibility_location_base}/{t_name}",

}

 

However, at runtime I get the following error:

 

Input path url 'abfss://xxx@accountname.dfs.core.windows.net/bronce_name_location/use_case/table_name' overlaps with other external tables or volumes within '' call. Conflicting tables/volumes: bronce_name_location/schema/table_name.

 

If I remove the table_properties, the streaming table is created successfully.

After that, I tried to alter the table using SQL to add the external location, but I get an error indicating that the table is managed by the pipeline and must be modified through it.

 Questions

1. Is it possible to associate an external location with a table created via an SDP (DLT) pipeline?
2. If so, what is the correct way to configure this without causing the overlap conflict?
3. Are there recommended patterns for exposing DLT-managed tables to external systems like Snowflake?

[Brahmareddy]

Hi ,

How are you doing today? as per my understanding, DLT fully manages the tables it creates, so you can’t attach an external location directly to them. That’s why you’re seeing the overlap error.

Instead of trying to convert the DLT table into an external table, keep it managed and expose it separately. The cleanest option is Delta Sharing if Snowflake supports it. Otherwise, create a downstream external table that reads from the DLT table and writes to a different location.

Don’t override what the pipeline owns. Expose it through a supported pattern.

Hope this helps! let me know for any additional questions.

Regards,

Brahma

[Jotaefe1991]

Hi Brahma,

Thanks for the reply, I wanted to implement the Compatibility Mode referenced in this link

https://docs.databricks.com/aws/en/external-access/compatibility-mode

“Using Compatibility Mode, you can read Unity Catalog managed tables, materialized views, and streaming tables from external systems while maintaining optimal performance on Databricks. This feature automatically generates read-only versions of your tables that can be accessed by any Delta Lake or Iceberg client.”

Do you have any insights on this one?

[ShamenParis]

Hi @Jotaefe1991 ,

The overlap error you are hitting is a Unity Catalog storage collision, not a DLT limitation.

Here is exactly what is happening and how to fix it:

The path you provided for "delta.universalFormat.compatibility.location" (abfss://.../bronce_name_location/...) is overlapping with an existing Volume, External Table, or the managed storage path of the schema itself. Unity Catalog strictly prevents multiple entities from managing the same storage path to avoid data corruption.

To expose this DLT table via Compatibility Mode without conflicts, you need to write the compatibility metadata to a dedicated, isolated path.

1. Create a new, dedicated path in your ADLS container strictly for compatibility metadata (e.g., abfss://xxx@accountname.dfs.core.windows.net/compatibility_metadata/).

2. Register this new path as an External Location in Unity Catalog (and ensure the DLT pipeline's cluster/identity has write permissions to it).

3. Update your DLT decorator to point to this clean path:

table_properties={
    "delta.universalFormat.enabledFormats": "compatibility",
    "delta.universalFormat.compatibility.location": "abfss://xxx@accountname.dfs.core.windows.net/compatibility_metadata/use_case/table_name",
}

Once you point the compatibility location to a distinct, non-overlapping path, DLT will successfully create the streaming table and maintain the read-only Iceberg/Delta formats for Snowflake!