Here's your Data + AI Summit 2024 - Data Governance recap as you navigate the explosion of AI, data and tools in efforts to build a flexible and scalable governance framework that spans your entire data and AI estate. Keynote: Evolving Data Governan...
Is it possible to let a group in databricks to change a delta table, but not the table’s schema in UC? i.e., they can add or remove rows of the delta table, but cannot change the name or the format of the columns.
Hi @NW1000, No, Unity Catalog doesn’t let you separate change rows from change schema that finely. To let a group insert/update/delete rows on a table, you must give them MODIFY on that table. MODIFY also allows ALTER TABLE operations like ADD/DROP/...
Hi,I want to import Databricks metadata information from unity catalog including lineage into GCP Dataplex governance solution. Can you let me know the how this can be achieved.Similar to Purview and Unity Catalog integration, are there APIs to push ...
Hi Walter,Is it possible to migrate Databricks Unity Catalog data from Databricks to Google Dataplex without using API calls? What's the efficient way to move Unity Catalog data from Databricks to Google Dataplex? Please explain the steps. Thank you ...
Hi everyone,I’m curious if anyone has successfully implemented Databricks Genie (chat/agent) for production use.Currently, we’ve enabled a few Genie instances for power users who are comfortable working with data outside of the data team. However, we...
Hi Julie, thanks for sharing the screenshot. There are a few things we'd recommend to customers when they are starting out on a Genie journey, we defintiely see enterprises creating Genie spaces at scale but it needs some thought and guardrails. Some...
I have a Python-based FastAPI application with a React frontend, which I would like to deploy to Databricks apps. I'm able to do it, but having trouble with file system access. In addition to database access, my application relies on some file operat...
In the end, I just added the wheel to the app's source code and installed from there. Not ideal, but the module is small and checking the wheel into git was the lesser issue.This workaround runs fine for a few months now. My build pipeline takes care...
How is Unity Catalog managed in real time at an enterprise scale, including workspace-level restrictions, privilege-based ACLs, row- and column-level security, ABAC, and tag-driven governance, and which languages or tools are used to manage the entir...
Thank you... Which roles ( workspace admin or Metastore Admin) are recommended to securely create and manage Unity Catalog objects (Storage Credentials, External Locations, Catalogs, Schemas, and Delta Sharing)?” and why ?
Best Practices for Securing Access to Data in DatabricksUnity Catalog is the unified governance solution for Data & AI assets in Databricks and greatly simplifies and centralized data access control. This guide includes best practices for both the st...
Which roles ( workspace admin or Metastore Admin) are recommended to securely create and manage Unity Catalog objects (Storage Credentials, External Locations, Catalogs, Schemas, and Delta Sharing)?” and why ?
The company that I work for has DDL where the column names/table names are not always that intuitive. It would be great to see the "Title" clause such that is used in ANSI SQL, added to the DDL statement so that the logical name can come through to t...
Hi Sandy, the best way of getting this as a feature request is to go through your account team, that way the weight of your account is behind the feature request and it's more likely to be prioritised. I've had a look internally and I can't find this...
In the Databricks one demo, and even in official website I seehttps://www.databricks.com/blog/introducing-databricks-one"Domains", how to enable it ? Or add "Domain" to data?
Hey, I've seen that solution is already available, thanks for remindind.
Hi,I am currently designing a PII governance framework to meet CCPA compliance requirements on Databricks. I understand that Databricks provides mechanisms such as VACUUM and Deletion Vectors combined with REORG … APPLY (PURGE) to permanently remove ...
Hi @abhijit007 , A new Data Classification feature (currently in Public Preview), allows to automatically classify and tag sensitive data in your catalog. It goes through few steps: AI-driven engine scans Unity Catalog tables and detects PII data and...
I am new to Databricks and trying to understand the concepts first, before getting into technical deepdive. I am a bit confused between Role Based Access Control and Unity Catalog. Both feature controls access to the data. Can someone provide me the ...
I have created custom tags on a column and plan to mask columns with tags via policy. I am facing 2 issues 1. Cant see the custom tag under - Mask column if it has specific tag. 2. If I type my custom tag get error when creating policy Policy creat...
Hi, to use a tag in a tag policy it needs to be a governed tag rather than just a general tag. IF you just create it using Set tags UC sees it as a informational tag rather than a governed tag. If you use a CREATE tag statement to create it then you'...
We are building a cost monitoring dashboard and want to drill down from pipeline level cost to individual table or flow cost inside a multi table DLT pipeline.Our setup is a serverless DLT pipeline with multiple tables including more than ten streami...
Hello @Louis_Frolio , Thank you for your detailed response and validating my approach. This comment has been very insightful. I highly appreciate it. I believe I know what I need for now. Regards,Reema
Hello Databricks Community,Databricks is making great progress towards becoming a truly governed and unified catalog. When it comes to governing and cataloguing physical data assets, it already offers a strong foundation. With the introduction of met...
Hello @MauriceDekker , Good question, and you've framed it well — UC has genuinely gotten strong for physical data assets and metrics, so reports and business terminology feel like the obvious next frontier. Here's how I see customers approaching th...
While using DQX to generate DQ rules on a table i have used the following prompt.from databricks.labs.dqx.profiler.generator import DQGeneratorfrom databricks.labs.dqx.profiler.profiler import DQProfilerfrom databricks.labs.dqx.config import InputCon...
I have granted table level permissions in Unity Catalog for users but users are still facing access issues when PowerBI reports are using these tables. Is this expected or I am missing some configuration setting?
Hi @Addyy, @balajij8 is right. For Power BI (via a SQL warehouse), users need a full chain of Unity Catalog + warehouse permissions. If you’ve only granted SELECT at the table level to individual users, but not USE CATALOG/USE SCHEMA on the hierarchy...
