Here's your Data + AI Summit 2024 - Data Governance recap as you navigate the explosion of AI, data and tools in efforts to build a flexible and scalable governance framework that spans your entire data and AI estate. Keynote: Evolving Data Governan...
I have created custom tags on a column and plan to mask columns with tags via policy. I am facing 2 issues 1. Cant see the custom tag under - Mask column if it has specific tag. 2. If I type my custom tag get error when creating policy Policy creat...
Hi, to use a tag in a tag policy it needs to be a governed tag rather than just a general tag. IF you just create it using Set tags UC sees it as a informational tag rather than a governed tag. If you use a CREATE tag statement to create it then you'...
We are building a cost monitoring dashboard and want to drill down from pipeline level cost to individual table or flow cost inside a multi table DLT pipeline.Our setup is a serverless DLT pipeline with multiple tables including more than ten streami...
Hello @Louis_Frolio , Thank you for your detailed response and validating my approach. This comment has been very insightful. I highly appreciate it. I believe I know what I need for now. Regards,Reema
While using DQX to generate DQ rules on a table i have used the following prompt.from databricks.labs.dqx.profiler.generator import DQGeneratorfrom databricks.labs.dqx.profiler.profiler import DQProfilerfrom databricks.labs.dqx.config import InputCon...
I have granted table level permissions in Unity Catalog for users but users are still facing access issues when PowerBI reports are using these tables. Is this expected or I am missing some configuration setting?
Hi @Addyy, @balajij8 is right. For Power BI (via a SQL warehouse), users need a full chain of Unity Catalog + warehouse permissions. If you’ve only granted SELECT at the table level to individual users, but not USE CATALOG/USE SCHEMA on the hierarchy...
HelloLooking for guidance to keep data masked if it needs to link with unmasked data, or analysis requires a combination of masked and unmasked data. How have you tackle this challenge?Thank you.
Thank you for responding. It definitely gives us some thought of how far to extend "data mesh" principles in the physical data layer.
While learning about Databricks foreign catalogsn (I'm on free tier DBX account), seems theres 2 ways creating foreign catalog to Snowflake:via CONNECTION type=snowflake, seems jdbc connection through where DBX pulls all metadata of 1 snowflake datab...
Hi @emanueol, Your follow-up question is clear, and it is a good distinction to make. Let me address it directly. SHORT ANSWER Yes, when you set up Snowflake Catalog Federation in Databricks, Databricks does use the Snowflake Horizon REST Iceberg cat...
Hi,I cannot find so far a way to get programmatically (SQL/Python) the Subqueries(/Sub-statements) executions history records, shown in ADBricks UI Query History/Profile, that were executed during a TaskRun of Job, as shown in [red boxes] on the atta...
Hi @ADBricksExplore, The sub-statements panel you see in the Query History UI (the child queries spawned by a CALL or multi-statement execution) is rendered from an internal GraphQL endpoint that is not part of the public Databricks API surface. That...
I'm new to Azure Databricks and I'm facing an issue when trying to create a schema or table that points to my Azure Storage account. I keep getting this error:```[EXTERNAL_METASTORE_CLIENT_ERROR.OPERATION_FAILED] Client operation failed: org.apache.h...
Hi @cobba16, This is a common scenario on Azure Databricks when using the legacy hive_metastore with external storage. The root cause is that dbutils.fs.ls and Spark read operations use the credentials you set in the Spark session config, but the Hiv...
We are encountering issue lately around 17 January 2026.We make a Pipeline using DLT and RLS for filtering rows based on user's email. Previously our Row Level Security (RLS) is working fine. Until 3 days ago the dashboard is got into problem said: ...
Hi @ikhwan, Based on your description, you are seeing two separate error messages when querying tables that have row filters applied: 1. "Query on table with row filter or column mask not supported on assigned clusters" 2. "INVALID_PARAMETER_VALUE: D...
Hi,I’m trying to use Databricks Asset Bundles (DAB) to assign Unity Catalog grants to catalogs and schemas that already exist in my workspace.These catalogs and schemas were not originally created through DAB, but I would now like to manage their gra...
Hi @vjussiiih, Let me walk you through this. You are correct that DABs treat schemas and catalogs defined under "resources" as fully managed resources, which means they attempt to create them on first deploy and manage their full lifecycle. This is w...
Hi everyone,I’m currently testing the Data Classification feature in Databricks Unity Catalog.On the dashboard after classification completes, there’s a column called **“User Access.”** Based on the description, it seems to show the number of users w...
Hi @r_w_, Appreciate you sharing the details. the "User Access" column in the data classification results view is one of those features that is not thoroughly covered in the public documentation yet, so I understand the confusion. Here is what I have...
Hi Team,Hope you guys are doing well. Question 1:I am using this Databrick unity catalog rest api endpoint api/2.0/lineage-tracking/table-lineage?table_name=schemaname.catalogname.tablename&include_entity_lineage=true for getting lineage . Is this of...
Thank you so much for response and solution in details.I have one question in Unity Catalog lineage UI showing last 18 months also, but as per official documentation Lineage retains up to last 1 Year. Why this contradicts statement! Also, I am trying...
Is there a current way to limit data exporting. Meaning not allowing users export data from a db objects to csv, excel, or copying results to clipboard. I see that currently exporting/downloading data can be done at workspace level, but I am proposin...
Hi @bigdatabase, As of today, Databricks lets you control downloading notebook results only at the workspace level. As you pointed out, a workspace admin can disable downloading results from the notebook for all users in that workspace. However, ther...
We are attempting to create a comprehensive Data Governance solution using Unity Catalog & INFA CDGC tool. The objective is to onboard the business assets on the Informatica DG platform and use the unity catalog to trace technical assets and lineage ...
@TanushM -- We are also planning to do the same. However we are struggling with the volume of tables inside some schemas in a catalog. Can you suggest some of the best practices you have come across while integrating Unity catalog to CDGC?
It is our understanding that enabling unity catalog means that some metadata (for example usernames) will be sent to the USA for processing/storage. This is unfortunately a deal-breaker for us, and we need the data to reside solely in Europe for comp...
Usernames have always been stored centrally in the US. That's similar to most global services for the IAM Layer. The only thing additional that gets stored in the US as part of UC is the metastore names. you can check this link ( https://azure.micros...
