topic Re: Databricks API - Create Connection in Data Engineering

Databricks API - Create Connection

giladba — Mon, 08 Jul 2024 10:02:48 GMT

Hi,

Is it possible to use the Databricks API to create a connection to a different Azure Databricks workspace?

Thanks

Re: Databricks API - Create Connection

jacovangelder — Mon, 08 Jul 2024 10:14:15 GMT

No. You can use either Unity Catalog, Databricks Connect, Delta Sharing or DBFS mounts to access data in another databricks workspace (with its own cloud storage)

Re: Databricks API - Create Connection

giladba — Mon, 08 Jul 2024 10:28:30 GMT

It is possible to create a connection and create a foreign catalog which uses this connection, through the Databricks UI . For some reason, the Databricks API doesn't seem to be able to handle creating the connection to a different Databricks workspace.

Re: Databricks API - Create Connection

jacovangelder — Mon, 08 Jul 2024 10:45:03 GMT

Ah you mean a connection inside Unity Catalog, that wasn't really clear from the starting post.
I've done all of this with Terraform, so there must be an API available for it. Here it is: https://docs.databricks.com/api/azure/workspace/connections/create.

For creating a Databricks to Databricks connection, you'll have to either set Unity Catalog permissions or remove the catalog isolation in the workspaces.

Re: Databricks API - Create Connection

giladba — Mon, 08 Jul 2024 10:55:24 GMT

My apologies for not being cleared in the first post.

Terraform can handle it. The API is probably behind and can't handle the connection to Databricks yet. The best solution would probably be to just move to Unity Catalog...
Thanks for your quick reply