- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-05-2024 10:23 AM - edited 06-05-2024 10:24 AM
When I'm running `dbt run -s model` for different models in parallel on databricks general compute cluster. Im getting the error: TABLE_OR_VIEW_ALREADY_EXISTS. Also, at the same time, the view/table was not created at all.
But I run them in sequence, then no such error will show.
Any idea what is causing this?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-06-2024 12:54 PM
Hi @claudiazi,
It's hard to determine the exact root cause of the issue, but there's a known issue that could lead to the same error.
If you check CREATE TABLE [USING] - Parameters. You'll be able to see the following note:
Databricks strongly recommends using REPLACE instead of dropping and re-creating Delta Lake tables.
This is because the pattern of deleting and creating Delta Lake tables without the REPLACE clause might not clean-up all the metadata files right away.
Raphael Balogo
Sr. Technical Solutions Engineer
Databricks
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-07-2024 06:34 AM
@raphaelblg thaanks a lot for your reply! However the command were `create or replace view/table` Thats why I'm very surprised that we are getting this error message 🤔🤔🤔
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-07-2024 07:10 AM
@claudiazi Are you the owner of the view?
Raphael Balogo
Sr. Technical Solutions Engineer
Databricks
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-07-2024 07:44 AM
@raphaelblg yes! Im the owner. However, these views are inside the hive_metastore catalog. Could it be the reason?
Many thaaanks!
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-07-2024 08:06 AM
@claudiazi from what we've discussed so far I see no reason for the failure.
Not sure if this error is something the community can help with but maybe opening a support ticket through https://help.databricks.com/s/ might be a good idea.
Where's this query being submitted? Interactive cluster, SQL Warehouse, dbt?
Raphael Balogo
Sr. Technical Solutions Engineer
Databricks
