cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

Loading tables to gold, one loads and the other two fail but same process.

190809
Contributor

Hi team, I am still fairly new to working with delta tables. I have created a df by reading in data from existing silver tables in my lakehouse. I read in the silver tables usiung sql into a workbook, do some manipulation, unnest some fiels and then attempt to load as gold dlt tables. One of my gold tables works and the other two fail with the error 'cannot create ... because managed table already exists'. Any tips, tricks, advice on how to handle this. My dlt load looks like this:

<@dlt.table(comment='gold_users processed from silver platform users table to create gold tables ready for aggregation.'

)

def gold_users():

return users_df/>

1 ACCEPTED SOLUTION

Accepted Solutions

190809
Contributor

Hi @Pravin Chaubeyโ€‹ thanks for responding. I discovered the issue. I had to load them as unmanaged tables but had previously not specified a path when doing .saveAsTable() and so those two tables that were failing to load were in fact managed tables as you said.

View solution in original post

2 REPLIES 2

ChaubeyPravin
New Contributor II

Hi ,

It seems like there is already a managed table with same name.

If you want to anyhow write you need to remove first exist table with same name or if your schema is same you can append data to table.

To remove existing table you can use dbutils.fs.rm('path',True)

190809
Contributor

Hi @Pravin Chaubeyโ€‹ thanks for responding. I discovered the issue. I had to load them as unmanaged tables but had previously not specified a path when doing .saveAsTable() and so those two tables that were failing to load were in fact managed tables as you said.

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโ€™t want to miss the chance to attend and share knowledge.

If there isnโ€™t a group near you, start one and help create a community that brings people together.

Request a New Group