topic Why catalog API does not include the catalog ID in the response? in Administration & Architecture

Why catalog API does not include the catalog ID in the response?

littlewat — Mon, 12 May 2025 09:03:37 GMT

Hi!

I'm using Terraform(TF) to manage the Databricks resources.

I would like to rename the Unity catalog using TF, but I could not. (similar issues have been reported for this:
- https://github.com/databricks/terraform-provider-databricks/issues?q=is%3Aissue%20state%3Aopen%20rename%20catalog )

After investigating this issue, I found that the Terraform databricks_catalog resource is using the catalog name as its ID.
https://github.com/databricks/terraform-provider-databricks/blob/1982317e41702b4210e54666a5681cfd2b1b8c92/catalog/resource_catalog.go#L87

This causes several serious problems, including:

It's impossible to rename a catalog using Terraform.
If I rename a catalog created by Terraform via the UI, Terraform will recognize it as deleted and recreate it based on theTerraform configuration. As a result, two catalogs will exist.

I looked into why Terraform is using the name as the ID, and it turns out that the REST API does not include the actual catalog ID in its response.
https://docs.databricks.com/api/workspace/catalogs/get

Could you please modify the REST API to include the catalog ID in its response?
This issue is causing us significant trouble.

Thank you!

Re: Why catalog API does not include the catalog ID in the response?

Louis_Frolio — Tue, 13 May 2025 14:00:23 GMT

The issue regarding renaming a Unity Catalog using Terraform is rooted in the way the Terraform databricks_catalog resource is implemented. Specifically, the catalog name is being used as the resource's ID, which is a direct consequence of limitations in the Databricks REST API response that does not include an actual catalog ID. Because of this:

It is currently not possible to rename a catalog via Terraform.
Renaming a catalog outside Terraform (e.g., via the UI) causes Terraform to interpret the catalog as deleted and recreate it based on the Terraform configuration, leading to duplicate catalogs.

From your investigation, it seems these behaviors align with certain known limitations within the Databricks Terraform provider. The relevant code snippet in resource_catalog.go confirms the use of the name as the ID:


Id: catalog.Name,

You have correctly identified the root cause: the Databricks REST API does not provide a unique catalog ID in its response for workspace catalog operations. Although the Terraform provider relies on the API, this limitation is outside the scope of Terraform itself.

Hope this helps, Lou.

Re: Why catalog API does not include the catalog ID in the response?

littlewat — Wed, 14 May 2025 00:45:50 GMT

@Louis_Frolio Thank you for your quick and thorough response! I'm glad to know that my assumption was correct.

Would it be possible for you to escalate this issue to the backend team that is developing the catalog REST API?

Re: Why catalog API does not include the catalog ID in the response?

Louis_Frolio — Wed, 14 May 2025 13:41:04 GMT

I will pass your request along; however, there is nothing I can do to escalate the issue. I can only make the request. Cheers, Lou.