โ06-05-2024 10:23 AM - edited โ06-05-2024 10:24 AM
When I'm running `dbt run -s model` for different models in parallel on databricks general compute cluster. Im getting the error: TABLE_OR_VIEW_ALREADY_EXISTS. Also, at the same time, the view/table was not created at all.
But I run them in sequence, then no such error will show.
Any idea what is causing this?
โ06-07-2024 08:06 AM
@claudiazi from what we've discussed so far I see no reason for the failure.
Not sure if this error is something the community can help with but maybe opening a support ticket through https://help.databricks.com/s/ might be a good idea.
Where's this query being submitted? Interactive cluster, SQL Warehouse, dbt?
โ06-06-2024 12:54 PM
Hi @claudiazi,
It's hard to determine the exact root cause of the issue, but there's a known issue that could lead to the same error.
If you check CREATE TABLE [USING] - Parameters. You'll be able to see the following note:
Databricks strongly recommends using REPLACE
instead of dropping and re-creating Delta Lake tables.
This is because the pattern of deleting and creating Delta Lake tables without the REPLACE clause might not clean-up all the metadata files right away.
โ06-07-2024 06:34 AM
@raphaelblg thaanks a lot for your reply! However the command were `create or replace view/table` Thats why I'm very surprised that we are getting this error message ๐ค๐ค๐ค
โ06-07-2024 07:10 AM
@claudiazi Are you the owner of the view?
โ06-07-2024 07:44 AM
@raphaelblg yes! Im the owner. However, these views are inside the hive_metastore catalog. Could it be the reason?
Many thaaanks!
โ06-07-2024 08:06 AM
@claudiazi from what we've discussed so far I see no reason for the failure.
Not sure if this error is something the community can help with but maybe opening a support ticket through https://help.databricks.com/s/ might be a good idea.
Where's this query being submitted? Interactive cluster, SQL Warehouse, dbt?
Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโt want to miss the chance to attend and share knowledge.
If there isnโt a group near you, start one and help create a community that brings people together.
Request a New Group