Databricks Community

Datazilla · ‎11-14-2023

I am working at large company with many more or less independent divisions and we are currently working on the roll out of Unity Catalog in Azure. The idea was to have a central infrastructure repository (deployed via Terraform) to manage all central components like the Databricks Account and the UC metastore. We also wanted to create the UC catalogs here to enforce a specific naming conventions and other standards like tagging etc. The creation of resources within the catalogs is then up to the respective catalog owners. The root Storage Accounts (one per catalog) have also network restriction which requires to allow the networks of corresponding bound workspaces

This month Automatic enablement of Unity Catalog was announced which automatically enables new workspaces for UC. Furthermore, the workspace admins will automatically get the permission on the metastore to create UC catalogs. With this new behaviour we can no longer enforce our central catalog standards.

How do you deal with this situation? Do you also centrally manage all Databricks Workspaces to have full control of all Workspace Admins? It would be great to configure the permissions of workspaces admins in the Account console.

Datazilla · ‎11-23-2023

@Retired_mod wrote:
Workspace Admins: Consider configuring permissions for workspace admins in the Account console to strike a balance between autonomy and governance.

Is there such a configuration in the Account Console? The automatic enablement is rolled out sequentially and our Account is not migrated yet.

SSundaram · ‎11-30-2023

Without an option to enable/disable the auto creation of catalogs on the account level, this feature can/will never support "Central management" and also causes unnecessary tailwinds for organizations which have been on central governance and a new workspace is created. I prefer the way it was before, workspaces and catalogs just binded. That way it supported all forms of governance.

Datazilla · ‎11-30-2023

I totally agree.

In our central management we create a dedicated Azure Storage Account for each Catalog. Depending on the Catalogs isolation mode only specific Workspaces have network access to the Storage. The root storage of the Metastore is completely blocked. This means the automatically or de-centrally created Catalogs could not even be used to storage managed data due to missing network access.

Datazilla · ‎11-30-2023

I totally agree.

In our central management we create a dedicated Azure Storage Account for each Catalog. Depending on the Catalogs isolation mode only specific Workspaces have network access to the Storage. The root storage of the Metastore is completely blocked. This means the automatically or de-centrally created Catalogs could not even be used to storage managed data due to missing network access.

Datazilla · ‎02-12-2024

Workspace Admins: Consider configuring permissions for workspace admins in the Account console to strike a balance between autonomy and governance.

@Retired_mod Do you have any information about this configuration? I cannot find such thing in the Account Console. (In my opinion your answer looks LLM generated. So it could be hallucination. If it is not generated, I am sorry)

The automatic enablement for UC has not been rolled out to our account yet.