DLT pipeline table already exists error

Mukul3012
New Contributor II

‎02-10-2025 08:15 PM

Hi All,

I have been facing an issue with few of my DLT pipelines.

source code:

CREATE OR REFRESH STREAMING TABLE ****
TBLPROPERTIES(
    "enableChangeDataFeed"= "true",
  "delta.autoOptimize.optimizeWrite"="true"
)


AS SELECT *,
      _metadata.file_path as filepath,
      current_timestamp as load_date_time
  FROM cloud_files("abfss://*********@**********.dfs.core.windows.net/*******/******", "csv", map("inferSchema", "true", "mergeSchema", "true"))

 

I run this pipeline on a daily basis which uses autoloader to refresh my streaming table, it is reading in parquet format.

However, this pipeline throws an error:

Update 2c50a6 has failed while trying to update table 'cm-dev-upark-adb-001.bronze.__materialization_mat_*******************************.

 

com.databricks.sql.managedcatalog.UnityCatalogServiceException: [RequestId=***** ErrorClass=TABLE_ALREADY_EXISTS.RESOURCE_ALREADY_EXISTS] Table '*******' already exists

any ideas as to what might be the issue? I have a few dlt pipeline which use autoloader to query files from the same container withtin that storage account but some run fine and some throw this error.

 

0 Kudos

Alberto_Umana
Databricks Employee
Databricks Employee

‎02-11-2025 04:58 AM

This can happen if the table was not properly dropped or if there is a naming conflict.

Before creating or refreshing the table, check if it already exists in the catalog. 

SHOW TABLES IN <database_name>;
0 Kudos

Mukul3012
New Contributor II
In response to Alberto_Umana

‎02-11-2025 02:34 PM

it did not come up using that command nor in the catalog explorer.

I created a new dlt pipeline and then pointed it to the same notebook which seems to have solved the problem.

Is this a known bug with dlt?

0 Kudos