Here's your Data + AI Summit 2024 - Data Governance recap as you navigate the explosion of AI, data and tools in efforts to build a flexible and scalable governance framework that spans your entire data and AI estate. Keynote: Evolving Data Governan...
Hi everyone, We're considering bringing a section called "For You" in the catalog tree view into the Catalog Explorer tab. Right now this section surfaces active tables and favorited UC objects in authoring surfaces (SQL Editor, Notebooks, and Dashbo...
That makes a lot of sense! We're considering revamping it to not have the For You pills and to put the favorites up at the top
I am using Azure Databricks DBFS. I have stored some python scripts, notebooks in folders in DBFS root and have mounted ADLS Gen2 to DBFS. Currently I do not have Unity Catalog enabled and this was setup 3-4 years ago. I came across this article http...
Hello @humayun! DBFS mounts and root storage are deprecated, as they are not secure with the Unity Catalog and pose security risks. While there’s no official end-of-life date, Databricks discourages using mounts and root storage. Existing setups will...
We have many suscription with databricks workspaces active, but i cant create a metastore. The button is inactive.
Hi @DiegoRU ,Thank you for providing the details. You also need to have the role of global administrator on the azure subscription To create metastore, please login to this link https://accounts.azuredatabricks.net/login as admin and you should be ab...
Hi,we are trying to move some of our code from a ‘legacy’ cluster to a ‘Multi-node/ Shared' cluster so that we can start using Unity Catalog. However, we have run into an issue with some of our code, which calls Stored Procedures, on the new cluster....
We face the same issue. Our code depends heavily on stored procedures (best practice, no).Unfortunately, when moving to shared clusters and DBR 13.3, this does not work anymore.driver_manager = spark._sc._gateway.jvm.java.sql.DriverManager connection...
Hi,I deployed a Unity Catalog metastore some time back (screenshot below).But when I am trying to create a new one, the create metastore option is disabled. Hovering on the button shows the message: "You must have an active subscription to create a M...
Are you using Azure Databricks ?? have you recently migrated your classic Global Administration role to RBAC global admin in azure we had the same issue we fix it by creating metastore from CLI databricks metastores create {Name} --region {regi...
As written here: Compute access mode limitations for Unity Catalog | Databricks on AWSWhy is R language not supported?
Is there any update on when the timeline will come out? I am unable to use a shared cluster at work because shared computes do not have R as a supported language. Thanks
Hey Guyz, We need to scan the complete schema present at unity catalog to an external user or group of users without onboarding them on to our platform. Is there a way we can expose this details to them. Additionally, can we expose the sample dataset...
Hi @RohitKumar7 , Greetings! Looking at your request, i would like to confirm you that it would be possible to use the Delta sharing feature. Delta sharing feature lets you share data and AI assets with users outside your organization, whether or no...
Hello,I would like to know if there is direct access to the Databricks query history tables. For compliance issues, I would like to be able to create reports for something like: who has accessed a particular column in a table in the past 6 months. T...
Throwing it out there that my team is also looking for a way to easily do this and I don't think the above solutions really fit our need. We are currently in the process of migrating schemas (from one name to a better name, we didn't rename because w...
Hi, as described in the title, when I try to explore the content of our metastore default location (external location created by default when the metastore is created) I get NO_PARENT_EXTERNAL_LOCATION_FOR_PATH. I've tried using a Serverless SQL ware...
The error NO_PARENT_EXTERNAL_LOCATION_FOR_PATH you are encountering is related to missing permissions for accessing the external location in Unity Catalog. This behavior is expected if the necessary permissions are not correctly set. Here are the ste...
Hi,I am trying to set up Unity-Catalog for my company and ran into a problem today. Basically, for each new source of data we ingest, we create a view-layer on top of the "tables". We do that because we have pseudonymized information in our datalake ...
Databricks actually has pretty good secrets management. If you need to secure a key, do this: https://learn.microsoft.com/en-us/azure/databricks/security/secrets/To use the key in your SQL statements, do this: https://learn.microsoft.com/en-us/azure...
Hi Community,I’ve been working on a data pipeline using Databricks Delta Live Tables (DLT) with Unity Catalog. My setup saves data into Azure Data Lake Storage (ADLS) for streaming tables. However, I’ve noticed that Unity Catalog encrypts various com...
Hi Community,I previously reached out regarding creating shortcuts in Microsoft Fabric for Databricks Delta Live Tables (DLT) managed through Unity Catalog, specifically when the data resides in Azure Data Lake Storage (ADLS) and appears encrypted.Af...
If I grant all privileges in my schema does that automatically give access to users for all underlying objects? Or should I give access seperately for all the objects?
Hi @Harikrish, In Unity Catalog, privileges are hierarchical and inherited downward. This means that granting a privilege on a catalog or schema automatically grants the privilege to all current and future objects within the catalog or schema. Theref...
Setup UCX from Databricks web terminal For cases when your desktop or laptop will not support the databricks labs install ucx technical requirements such as Python 3.10, administrative access, python package access or network access but your Databric...
I’ve found that using DBR 15.4, Personal Compute, and a Single Node is the most effective setup for installing UCX via web terminal. Note: The Databricks CLI is the recommended and most efficient option for managing the UCX installation. Always try t...
Hi,I want to import Databricks metadata information from unity catalog including lineage into GCP Dataplex governance solution. Can you let me know the how this can be achieved.Similar to Purview and Unity Catalog integration, are there APIs to push ...
To import Databricks metadata information from Unity Catalog, including lineage, into GCP Dataplex, you can use the Unity Catalog APIs to extract the necessary metadata and lineage information and then push this data into Dataplex using its APIs. Her...
When I try to interact with the "Browse DBFS" button, I get the following error: Operation failed: "The specified path, or an element of the path, exists and its resource type is invalid for this operation.", 409, PUT, https://dbstoragerrzvpacujrs22....
Hi @learning384e, Could you please share the command you are running including the action? does it have dbfs:/ in the path?
