Here's your Data + AI Summit 2024 - Data Governance recap as you navigate the explosion of AI, data and tools in efforts to build a flexible and scalable governance framework that spans your entire data and AI estate. Keynote: Evolving Data Governan...
Hi All,A question for those using Delta Sharing, how are you defining naming conventions for Shares and Recipient names? What best practices or standards are you following? Thanks, Fatima
For platform-to-platform environments, I nest all shared tables under a specially named Schema with a prefix of DS-. For example, my schema name could be ds-mfgschema. This is a 'flag' that any tables nested under this schema are being delta shared ...
I am exploring the search capabilities available in Unity Catalog. Specifically, I want to know if it is possible for a user to discover the existence of a table and its owner via search, even if they don't have access to the table's data. This featu...
I just found out that with the BROWSE privilege on the catalog this can be done
Due to some problems, I need to set Py4JSecurity to false, however I get the following message: "spark.databricks.pyspark.enablePy4JSecurity is not allowed when choosing an access mode"My cluster is in Shared access mode and a runtime version 12.1 (i...
Hello, any update on this? Is this still something not allowed in Databricks? I need to run a function that is not whitelisted and I need that function to run on a shared cluster as I need to read from a table that has a masking in it. So I need eith...
Dear community,Our company is using Databricks and we are happy to have Unity Catalog emerged to solve part of our Data Governance problems. We are very interested in the Table Insights feature which is newly introduced (View frequent queries and use...
Dear @Retired_mod, Thanks for your answer. I just checked and indeed such information can be queried to information_schema tables.Anyways, do you have the information if there exists a plan to expose such insights via REST API, or not at all?The reas...
Hi all,I am unable to search for tables in the search bar. I can see them under Catalog and I can even see them when browsing 'Location' in the search menu.What can be the issue here? I have tables both in the DBFS root and in a separate Storage Acco...
Hi and thank you for your reply.I can see that my workspace is attached to the metastore in the admin console. I am pretty new with Databricks but everything related to Unity Catalog looks good to me.Can you be more explicit about what you mean by "E...
Hi, Is it possible to track and display data lineage with Unity Catalog while using databricks on GCP? like so: https://docs.databricks.com/en/data-governance/unity-catalog/data-lineage.htmlIf it's not yet implemented, then is there any roadmap avai...
Assuming you wish to see the Lineage on a table/view in a UC Catalog, it's just a matter of navigating to Catalog Explorer, drill down to your UC Catalog>Schema>Table/View and clicking on the 'Lineage' tab. Note this functionality is not available in...
I transfered the owner of pre-created datasets (catalogs and schemas) by workspace admin to a service principle.ALTER CATALOG xyz OWNER TO `sp`;I think as the owner, the job owned by `sp`, should be able to create table in all schemas under catalog x...
Hi @hdu ,Below is an excerpt from documentation: "Owners of an object are automatically granted all privileges on that object. In addition, object owners can grant privileges on the object itself and on all of its child objects. This means that owner...
Hello, now that a metstore is created by default; I'd like to know where the corresponding storage account is. Is it hosted by Databricks? I'm guessing so; if that the case; from a compliance and security perspective; is it better to move it to a sto...
Hi,I got error [UC_NOT_ENABLED] Unity Catalog is not enabled on this cluster, when run in the notebook to create a catalog. Looks like I followed all the steps to set up workspace, metastore and compute. The steps are:create workspace from UI -> Qu...
Hi @Meghana_Vasavad - I followed these steps below to solve the similar issue I faced today.Evidently, your workspace is not UC enabled. Please find the MS Learn link below to follow the step-by-step instructions to enable UC via account console.http...
We are trying to add Slack as a "Notification destination" under "Workspace Settings" in our Databricks workspace.However, there is no option to actually grant permission on that destination to define who gets to actually use that destination. This i...
Hi @emmalily ,thanks for your answer. I am curious to hear more about your first proposed solution "creating private channels". I am not sure how that would prevent others from using the Slack notification destination since everyone in the Databricks...
In today’s data-driven landscape, organizations increasingly recognize the critical need for unified governance to ensure consistent data security, compliance, and efficiency. AWS Glue Data Catalog users seeking enhanced governance, operational contr...
Hello community,I got this error, I created a new recipient grant the share data and created a share data with the catalog, how can I proceed in the interested catalog, where do I need to assign USE_PROVIDER?
You need to ask a metastore admin to gran you the USE PROVIDER permission at the metastore level https://docs.databricks.com/en/data-governance/unity-catalog/manage-privileges/privileges.html#use-providerThis can be done through the UI, under Catalog...
I have created an external table in unity catalog and when describing its data getting the following errororg.apache.hadoop.fs.FileAlreadyExistsException: Operation failed: "The specified path, or an element of the path, exists and its resource type ...
Can you confirm you created a Storage Credential and External Location prior to the CREATE TABLE statement. Also can you post the CREATE TABLE syntax? Thanks....
Hi, I have a couple of questions 1) For Unity Catalog can we sync 2 Metastores which are in different regions?2) If latency is not an issue, can a single metastore handle 2 different regions (and their workspaces)
1. There is no way to sync 2 metastores specifically but you can use Delta Sharing to create a read only view of a source metastore on a target metastore. 2. Metastores are associated to the region, only workspaces in the same region can be assigned
In our Production tenant we have majority of our workspaces in the North Europe region. We have disaster recovery set up and these workspaces are in West Europe, so will require 2 MetastoresIf Unity Catalog was to get implemented, is there a way to s...
I dont see the reply form KanizCan you please share what was the reply you received to your question? Thanks in advanceSaurabh
