Here's your Data + AI Summit 2024 - Data Governance recap as you navigate the explosion of AI, data and tools in efforts to build a flexible and scalable governance framework that spans your entire data and AI estate. Keynote: Evolving Data Governan...
While learning about Databricks foreign catalogsn (I'm on free tier DBX account), seems theres 2 ways creating foreign catalog to Snowflake:via CONNECTION type=snowflake, seems jdbc connection through where DBX pulls all metadata of 1 snowflake datab...
Hi @emanueol, Your follow-up question is clear, and it is a good distinction to make. Let me address it directly. SHORT ANSWER Yes, when you set up Snowflake Catalog Federation in Databricks, Databricks does use the Snowflake Horizon REST Iceberg cat...
Hi,I’m trying to use Databricks Asset Bundles (DAB) to assign Unity Catalog grants to catalogs and schemas that already exist in my workspace.These catalogs and schemas were not originally created through DAB, but I would now like to manage their gra...
Hi @vjussiiih, Let me walk you through this. You are correct that DABs treat schemas and catalogs defined under "resources" as fully managed resources, which means they attempt to create them on first deploy and manage their full lifecycle. This is w...
Hi everyone,I’m currently testing the Data Classification feature in Databricks Unity Catalog.On the dashboard after classification completes, there’s a column called **“User Access.”** Based on the description, it seems to show the number of users w...
Hi @r_w_, Appreciate you sharing the details. the "User Access" column in the data classification results view is one of those features that is not thoroughly covered in the public documentation yet, so I understand the confusion. Here is what I have...
Hi Team,Hope you guys are doing well. Question 1:I am using this Databrick unity catalog rest api endpoint api/2.0/lineage-tracking/table-lineage?table_name=schemaname.catalogname.tablename&include_entity_lineage=true for getting lineage . Is this of...
Thank you so much for response and solution in details.I have one question in Unity Catalog lineage UI showing last 18 months also, but as per official documentation Lineage retains up to last 1 Year. Why this contradicts statement! Also, I am trying...
Is there a current way to limit data exporting. Meaning not allowing users export data from a db objects to csv, excel, or copying results to clipboard. I see that currently exporting/downloading data can be done at workspace level, but I am proposin...
Hi @bigdatabase, As of today, Databricks lets you control downloading notebook results only at the workspace level. As you pointed out, a workspace admin can disable downloading results from the notebook for all users in that workspace. However, ther...
We are attempting to create a comprehensive Data Governance solution using Unity Catalog & INFA CDGC tool. The objective is to onboard the business assets on the Informatica DG platform and use the unity catalog to trace technical assets and lineage ...
@TanushM -- We are also planning to do the same. However we are struggling with the volume of tables inside some schemas in a catalog. Can you suggest some of the best practices you have come across while integrating Unity catalog to CDGC?
It is our understanding that enabling unity catalog means that some metadata (for example usernames) will be sent to the USA for processing/storage. This is unfortunately a deal-breaker for us, and we need the data to reside solely in Europe for comp...
Usernames have always been stored centrally in the US. That's similar to most global services for the IAM Layer. The only thing additional that gets stored in the US as part of UC is the metastore names. you can check this link ( https://azure.micros...
While we evaluate moving our many autoloader configurations to use `cloudFiles.cleanSource` , we're wondering if we can instead just implement a lifecycle policy outside of Databricks that deletes files older than 30 days.Is there a problem with doin...
Hi why are you not planning to move away from directory listing mode to useManagedFileEvents?? execution will be faster and no more scanning of directories everytime. File events use a single Azure Databricks-managed file notification queue for all...
How is Unity Catalog managed in real time at an enterprise scale, including workspace-level restrictions, privilege-based ACLs, row- and column-level security, ABAC, and tag-driven governance, and which languages or tools are used to manage the entir...
Hi @APJESK ,The most common approach I've seen in enterprise is to use terraform to govern Unity Catalog. Below you can find a good series of articles that introduce this concept:https://pl.seequality.net/terra-dbx-p1/Databricks terraform provider is...
Hi,I cannot find so far a way to get programmatically (SQL/Python) the Subqueries(/Sub-statements) executions history records, shown in ADBricks UI Query History/Profile, that were executed during a TaskRun of Job, as shown in [red boxes] on the atta...
Greetings @ADBricksExplore , Short answer: there isn’t a supported public API that returns the “Substatements / Subqueries” panel you see in the Query History or Profile UI. The GraphQL endpoints the UI relies on are internal and not stable or suppo...
Hi everyone,I'm using the Budget Policy API (https://docs.databricks.com/api/account/budgetpolicy/create) to create Serverless budget policies. I can successfully create and retrieve policies, but I haven’t found any way to manage their permissions —...
Here are some helpful hints/tips/tricks: Programmatic Management of Budget Policy Permissions: Options and Best Practices 1. What is Possible Today? Yes, there is a programmatic way to manage permissions (user and group assignments) for Databricks Bu...
I’m trying to get a full list of Databricks workspace groups and their user memberships. I want to do this in two ways:As a queryable table or view (e.g., for audits, security reviews, app integration)From within a Databricks App (Streamlit-style), u...
@discuss_darende - you could use below code in the notebook.Pls adjust it based on your need.from databricks.sdk import AccountClient, WorkspaceClient # If env vars are set, this picks them up automatically a = WorkspaceClient() # List identities u...
We are encountering issue lately around 17 January 2026.We make a Pipeline using DLT and RLS for filtering rows based on user's email. Previously our Row Level Security (RLS) is working fine. Until 3 days ago the dashboard is got into problem said: ...
I have been trying to create a serverless workspace using us-east-1 and the RLS working fine. I think it's confirmed that only happened in specific region (in my case ap-southeast-1)
I'm new to Azure Databricks and I'm facing an issue when trying to create a schema or table that points to my Azure Storage account. I keep getting this error:```[EXTERNAL_METASTORE_CLIENT_ERROR.OPERATION_FAILED] Client operation failed: org.apache.h...
Hi Cobba16, A couple thoughts for you. It looks like you're setting the credentials in a spark config from within a Notebook. That's why operations execute in your session context, but when trying to create the Schema, the Hive Metastore is trying to...
If I grant all privileges in my schema does that automatically give access to users for all underlying objects? Or should I give access seperately for all the objects?
Are there Houston local practitioners interested in building up the data gov (strategy, implementation, value add) group? #unitycatalog #governedtags #systemtags #abac #principals #groups
