topic Re: Inquiry Regarding Enabling Unity Catalog in Databricks Cluster Configuration via API in Data Engineering

Inquiry Regarding Enabling Unity Catalog in Databricks Cluster Configuration via API

Himanshu4 — Thu, 23 May 2024 15:56:22 GMT

Dear Databricks Community,

I hope this message finds you well. I am currently working on automating cluster configuration updates in Databricks using the API. As part of this automation, I am looking to ensure that the Unity Catalog is enabled within the cluster configuration.

I understand that enabling Unity Catalog is typically done manually through the Databricks UI during cluster creation or modification. However, I would like to inquire if there is a way to programmatically enable Unity Catalog via the Databricks API.

In my current script, I am fetching job details and cluster configuration settings, but I am unable to reflect the enabled Unity Catalog in the cluster summary. I would greatly appreciate any insights or guidance on how to achieve this programmatically via the API.

Thank you very much for your assistance and support.

Re: Inquiry Regarding Enabling Unity Catalog in Databricks Cluster Configuration via API

raphaelblg — Mon, 27 May 2024 19:51:47 GMT

Hi @Himanshu4 ,

To run Unity Catalog workloads, compute resources must comply with certain security requirements. Non-compliant compute resources cannot access data or other objects in Unity Catalog. SQL warehouses always comply with Unity Catalog requirements, but some cluster access modes do not. See Access modes.

Re: Inquiry Regarding Enabling Unity Catalog in Databricks Cluster Configuration via API

Himanshu4 — Tue, 28 May 2024 06:46:33 GMT

Hi @raphaelblg ,

Thanks for your response. My task is to upgrade jobs. One of the scenarios involves upgrading job clusters, where I am trying to upgrade non-Unity Catalog clusters to Unity Catalog-enabled clusters. Is it possible by upgrading the spark version of the cluster using Databricks API?

Re: Inquiry Regarding Enabling Unity Catalog in Databricks Cluster Configuration via API

raphaelblg — Tue, 28 May 2024 22:03:10 GMT

@Himanshu4 as long as the job cluster is on Shared mode, UC should be accessible.

Re: Inquiry Regarding Enabling Unity Catalog in Databricks Cluster Configuration via API

Himanshu4 — Mon, 03 Jun 2024 05:24:16 GMT

Hi Raphael
Can we fetch job details from one workspace and create new job in new workspace with the same "job id" and configuration?

Re: Inquiry Regarding Enabling Unity Catalog in Databricks Cluster Configuration via API

raphaelblg — Fri, 20 Sep 2024 16:33:50 GMT

@Himanshu4 You can fetch the job details from one workspace but the job ID is auto-generated and non-customizable.