[Lakeflow Spark Declarative Pipelines] - Compatibility Mode not working

Jotaefe1991
New Contributor II

3 weeks ago

I’m working with an SDP pipeline that creates a streaming table using the dlt.create_streaming_table decorator. My goal is to expose this table through an external location so that a client can read it from Snowflake.

I attempted to configure this directly in the table definition by adding table_properties like this:

 
table_properties={
    "delta.universalFormat.enabledFormats": "compatibility",
    "delta.universalFormat.compatibility.location": f"{compatibility_location_base}/{t_name}",
}
 
However, at runtime I get the following error:
 
Input path url 'abfss://xxx@accountname.dfs.core.windows.net/bronce_name_location/use_case/table_name' overlaps with other external tables or volumes within '' call. Conflicting tables/volumes: bronce_name_location/schema/table_name.
 

If I remove the table_properties, the streaming table is created successfully.

After that, I tried to alter the table using SQL to add the external location, but I get an error indicating that the table is managed by the pipeline and must be modified through it.

 Questions

  1. Is it possible to associate an external location with a table created via an SDP (DLT) pipeline?
  2. If so, what is the correct way to configure this without causing the overlap conflict?
  3. Are there recommended patterns for exposing DLT-managed tables to external systems like Snowflake?
0 Kudos