Here's your Data + AI Summit 2024 - Data Governance recap as you navigate the explosion of AI, data and tools in efforts to build a flexible and scalable governance framework that spans your entire data and AI estate. Keynote: Evolving Data Governan...
Hi Community,We are working on implementing Databricks cluster policies across our organization and are seeking advice on best practices to enforce governance, security, and cost control across different environments.We have two main teams using Data...
Hi Team,I believe these are a few suggestions that can help!Start with environment-based policies: dev, qa, prodThese policies define the broadest guardrails (security, cost control, stability)Add team-specific variants only if requiredFor example: p...
We have a Delta Share that includes a partitioned table, and we want our recipient to be able to retrieve data only from a specific partition.I reviewed the Delta Sharing server documentation and found that it's possible to use SQL expressions for fi...
Hi @Col1ns ,All the things mentioned by @lingareddy_Alva are correct. Some additional things to consider. Check if your implementation of Delta Sharing supports predicate hints. Also, if your server supports them then your predicate should use restri...
I have nested JSON objects stored as values in some table columns. The table has a few top-level columns, and inside those, there are further nested JSON structures.When I write queries in the new SQL editor, the autocomplete correctly lists the firs...
@yit I could think of these possible reasons.1. Autocomplete relies on metadata to provide suggestions. Ensure the workspace is enabled for Unity Catalog and that the user has the necessary permissions to access the metadata for the tables containing...
Hi everyone, As part of our plans to improve the Discover Page, one of the next items we are looking into is building out Domains. 1. How have are your organizations using Data Domains today? 2. Can you share an example of your domain (and subdomain...
Hi @PeterWang,One of my clients in the financial services space groups their data using a domain-centric approach like customer, deposit, wealth, and credit.We tag tables and views with these domains and use that for RBAC, so access is granted based ...
When I run CREATE OR REPLACE VIEW on an existing view in Unity Catalog, the grants that were made on that object are removed. This seems like it is a bug. Is it on purpose or not?How to replicate:1. Create the viewRun the create or replace statement:...
Would also love any sort of communication back from Databricks on why create or replace view acts like this.I have had to run my CICD once with create (or replace) and then make a code change to swap to "alter view," which per https://docs.databricks...
Hi there,Is there any way to have an automatic Data Lineage without the usage of Unity Catalog, instead utilizing legacy Hive?My limited research shows there is no such option, at least without using some external tools which is not really an option ...
Hi @ty090 , you need to build a lineage generation tool if you have specific requirement of not using unity catalog or beyond databricks.
Hello all,We are currently running Azure Databricks notebooks through Azure Data Factory pipelines, where we use job clusters defined through policies. As an admin, I am able to see the list of all job clusters that have created when ADF makes a call...
Yes if you are run a notebook activity from ADF. You will not be able to see the job runs in databricks unless you are admin. But if you want to see any details you can use a python code to see the job runs and details of the run.You need a Service p...
I'm using the Databricks Lakehouse Monitoring API to enable monitoring across every table in a catalog. I wrote a script that loops through all schemas and tables and calls the create_monitor API for each one. However, when running the script from a ...
Hey all,We've got a bunch of business objects we created as delta live tables. these are important as that's what business users will use in dashboards, genie rooms etc. We're trying to enrich the metadata for those but the option is greyed out and I...
Solution that worked for me with materialized views created by DLT was to use the following syntax in SQL Editor in Databricks or run sql file with sql warehouse:COMMENT ON COLUMN <catalog>.<schema>.<table>.<column> IS 'comment here' This didn't work...
I have all the necessary permissions like USE CATALOG on Catalog, USE SCHEMA on Schema and SELECT,TABLE on Table. But still when I try to access the table from catalog I get the error : Insufficient privileges: User does not have USE CATALOG on Catal...
Hi, @SP_6721 and @Pawan_1 ,Thanks for your concerns. I was able to find the problem. So the problem was the cluster was assigned to groups and for the permissions were given to individual users. When the permissions were given to the group, we were a...
Hello,I tried to search for the topic in the community but couldn't find much around downstream lineage element that can be connected to a Unity Model.As far as I understood, if I have an inference workflow that loads a model from unity catalog and t...
Hello @FedeRaimondi, Downstream lineage: clear visibility from the model to the inference tables created by jobs or workflows consuming the model—is not supported currently in the same way as upstream lineage. I checked regarding this internally and...
Hi, Databricks CommunityI am currently plan to do migrate of existing metastore (not in a desired account and name) to a new one (a different desired account) within the same region. I understand it is not as straightforward it might be and complicat...
Hi team,I can't see Kaniz's recommendations. Are there any related links in this topic? Thanks&Regards,zmsoft
Having fun at the databricks conference
Hi all,I’ve been working with Unity Catalog in Databricks and using column masking policies for sensitive fields like SSNs. While trying to retrieve metadata about which columns have masking policies applied, I ran into some challenges and wanted to ...
@jv_v There are a few more approaches that you can try which are ideal for large scale scanning, Try using, Unity Catalog System Tables via system.information_schema Instead of <catalog>.information_schema.column_masksIf that doesnt work, use Databri...
Is there a command to hide the details of DLT/materialized views from users in UC?
Hi @dataminion01,As far as I know, Unity Catalog doesn't currently provide a direct way to hide DLT or materialized view definitions. Users with SELECT access can still view the logic via the "Details" tab.You can work around this by:Moving sensitive...
