[Lakeflow Spark Declarative Pipelines] - Compatibility Mode not working

[Jotaefe1991]

I’m working with an SDP pipeline that creates a streaming table using the dlt.create_streaming_table decorator. My goal is to expose this table through an external location so that a client can read it from Snowflake.

I attempted to configure this directly in the table definition by adding table_properties like this:

 

table_properties={

    "delta.universalFormat.enabledFormats": "compatibility",

    "delta.universalFormat.compatibility.location": f"{compatibility_location_base}/{t_name}",

}

 

However, at runtime I get the following error:

 

Input path url 'abfss://xxx@accountname.dfs.core.windows.net/bronce_name_location/use_case/table_name' overlaps with other external tables or volumes within '' call. Conflicting tables/volumes: bronce_name_location/schema/table_name.

 

If I remove the table_properties, the streaming table is created successfully.

After that, I tried to alter the table using SQL to add the external location, but I get an error indicating that the table is managed by the pipeline and must be modified through it.

 Questions

1. Is it possible to associate an external location with a table created via an SDP (DLT) pipeline?
2. If so, what is the correct way to configure this without causing the overlap conflict?
3. Are there recommended patterns for exposing DLT-managed tables to external systems like Snowflake?

[Brahmareddy]

Hi ,

How are you doing today? as per my understanding, DLT fully manages the tables it creates, so you can’t attach an external location directly to them. That’s why you’re seeing the overlap error.

Instead of trying to convert the DLT table into an external table, keep it managed and expose it separately. The cleanest option is Delta Sharing if Snowflake supports it. Otherwise, create a downstream external table that reads from the DLT table and writes to a different location.

Don’t override what the pipeline owns. Expose it through a supported pattern.

Hope this helps! let me know for any additional questions.

Regards,

Brahma

[Jotaefe1991]

Hi Brahma,

Thanks for the reply, I wanted to implement the Compatibility Mode referenced in this link

https://docs.databricks.com/aws/en/external-access/compatibility-mode

“Using Compatibility Mode, you can read Unity Catalog managed tables, materialized views, and streaming tables from external systems while maintaining optimal performance on Databricks. This feature automatically generates read-only versions of your tables that can be accessed by any Delta Lake or Iceberg client.”

Do you have any insights on this one?