topic Re: Setup unity catalog external location to minio in Administration & Architecture

Setup unity catalog external location to minio

gwilson — Thu, 26 Oct 2023 04:30:17 GMT

We have a minio server running in Azure that we have connected to the spark clusters directly. As we move to unity catalog, we would like to make the data stored in our minio servers accessible as an external location in Azure Databricks account via unity catalog. Is it possible to connect a non-ADLS object storage as an external storage location in Azure Databricks?

Re: Setup unity catalog external location to minio

174817 — Sun, 31 Mar 2024 08:03:51 GMT

Hi @Retired_mod ,

I have a server on Azure that supports the S3 protocol, and I am trying to follow these instructions in order to use Unity on Azure DataBricks with it. I am not sure about this part of your reply:

Set the Spark configuration values in the spark.conf file of your Azure Databricks workspace

I know how to configure Spark on a DataBricks Compute Spark cluster, but I could not find where to do this on a Unity catalog. I tried in the web GUI under "Catalog > Catalog Explorer > External Data" and also under "Compute / SQL Warehouses".

Is it possible to set a Unity Catalog external location to access an S3-compatible endpoint on Azure?

Thanks!

Re: Setup unity catalog external location to minio

gwilson — Sun, 31 Mar 2024 12:37:52 GMT

Support for S3-compatible storage is not available with unity catalog.