Here's your Data + AI Summit 2024 - Data Governance recap as you navigate the explosion of AI, data and tools in efforts to build a flexible and scalable governance framework that spans your entire data and AI estate. Keynote: Evolving Data Governan...
While we evaluate moving our many autoloader configurations to use `cloudFiles.cleanSource` , we're wondering if we can instead just implement a lifecycle policy outside of Databricks that deletes files older than 30 days.Is there a problem with doin...
Hi why are you not planning to move away from directory listing mode to useManagedFileEvents?? execution will be faster and no more scanning of directories everytime. File events use a single Azure Databricks-managed file notification queue for all...
Hi everyone,I'm using the Budget Policy API (https://docs.databricks.com/api/account/budgetpolicy/create) to create Serverless budget policies. I can successfully create and retrieve policies, but I haven’t found any way to manage their permissions —...
Here are some helpful hints/tips/tricks: Programmatic Management of Budget Policy Permissions: Options and Best Practices 1. What is Possible Today? Yes, there is a programmatic way to manage permissions (user and group assignments) for Databricks Bu...
I’m trying to get a full list of Databricks workspace groups and their user memberships. I want to do this in two ways:As a queryable table or view (e.g., for audits, security reviews, app integration)From within a Databricks App (Streamlit-style), u...
@discuss_darende - you could use below code in the notebook.Pls adjust it based on your need.from databricks.sdk import AccountClient, WorkspaceClient # If env vars are set, this picks them up automatically a = WorkspaceClient() # List identities u...
If I grant all privileges in my schema does that automatically give access to users for all underlying objects? Or should I give access seperately for all the objects?
Are there Houston local practitioners interested in building up the data gov (strategy, implementation, value add) group? #unitycatalog #governedtags #systemtags #abac #principals #groups
I am using Azure Databricks and Wanted to find out best way to enforce tagging to all the jobs (job compute / serverless jobs ) and follow standards any job not tagged should not run my all jobs are using asset bundle (running from github) please gui...
The easiest way to enforce tagging for jobs is to define a job cluster policy with custom_tags defined. Use this policy to create a job cluster for your production workloads. Any job that does not have the mandatory tags will fail as the job cluster ...
Is there a way to migrate a metastore to a different Databricks account in Databricks?If not, what alternative approaches can be considered? I can think of the following two:Using Delta Sharing to share objects from the source to the destination, and...
Databricks does not provide a built‑in way to “move” or migrate a Unity Catalog (UC) metastore from one Databricks account to another. Here is the list of activities that you can tryInventory & plan: Enumerate catalogs, schemas, tables (Delta vs. ext...
Can anyone share Databricks security model documentation or best-practice references
Sharing a few official Databricks security model and best-practice references that are widely used across enterprise implementations: Core Databricks Security ModelDatabricks Account & Workspace ArchitectureCovers account-level isolation, workspace b...
In my organization, we are using Databricks unity catalog and we have a metastore created for our region which holds all of our workspaces. When we created the metastore last year, we set a metastore root location for it (If I remember correctly, met...
In Databricks Unity Catalog, you’re right that you cannot simply “unset” the metastore root in place the way you would change a catalog setting. Historically, once a metastore root was defined, it stayed attached to the metastore.However, there is no...
Hello,Is SQL warehouses managed by Unity Catalog? My understanding is that since SQL Warehouse is part of the compute layer, Unity Catalog doesn't manage it as it only manages data layers.
@Senga98 SQL warehouses themselves are not governed by Unity Catalog, since they belong to the compute layer rather than the data layer. Unity Catalog focuses on managing data assets and how access to those assets is controlled, not on managing compu...
Hi!By default it seems users can only see tables and views in Unity Catalog that they have SELECT permission/privilege on. However, we would like to use Unity Catalog as a data catalog of tables we have. They wouldn't then be able to request access t...
@excavator-matt to make tables and views discoverable without exposing their data, you should use the BROWSE privilege at the catalog level.Granting BROWSE on a catalog (for example to the All-account users' group) allows users to view metadata acros...
We are currently managing our permissions via Terraform (including cluster creation, UC governance, etc.). We have a specific `data_engineer` role, and we need everyone with this role to be able to view and manage all of our SDPs.The Issue: Currently...
Our Solution: We moved job and pipeline permissions to DAB configuration files for streamlined enforcement. Terraform will remain the source of truth for workspace-level permissions only.
Hi All, We are in a process of migrating to UC from HMS and also changing the DBR from 11.3 to 16.4 LTS. When i am running sync schema from HMS to UC i am getting "Parquet does not support date. See HIVE-6384" error. Trying to resolve this, i have di...
@yashojhaIf you are doing HMS to UC migration. It is worth checking UCX an automated Unity catalog migration tool.https://github.com/databrickslabs/ucx
HelloIs SQL warehouses managed by Unity Catalog? My understanding is that since SQL Warehouse is part of the compute layer, Unity Catalog doesn't manage it as it only manages data layers.
Hi @Senga98 ,Your understanding is correct. Unity Catalog governs:Data objects (catalogs, schemas, tables, views, functions)Permissions (grants on the above)LineageGoverned storage locations & external locationsModel serving endpoints (UC Volumes / A...
I have an S3 account in which I have full administrator privileges. In that account I have a databricks workspace and an EFS filesystem setup. I created an interactive compute inside databricks workspace with the default config. How do I configure m...
if you’re trying to mount EFS directly to an interactive cluster, you’ll usually need to handle it through init scripts since EFS requires the NFS client to be installed and mounted at cluster startup. One thing to double-check is whether your worksp...
For some reason the SCIM API endpoint to get all the groups doesn't return any members information.According to the docs it should contain all members that are part of a group.Anyone know how to fix this?
Hi @ECCBV ,Which endpoint do you use? The following one will list for you all groups:List groups /api/2.1/accounts/{account_id}/scim/v2/Groups Now, if you want to list a members of the group you need to use following endpoint and provide group_id par...
