Here's your Data + AI Summit 2024 - Data Governance recap as you navigate the explosion of AI, data and tools in efforts to build a flexible and scalable governance framework that spans your entire data and AI estate. Keynote: Evolving Data Governan...
Hello, I have been trying to solidify my understanding of what changes occur after I upgrade my data to Unity Catalog. Our current ingestion process incrementally loads new data to tables in the hive metastore which are mounted to an s3 bucket. If I ...
Hi @Rachel Cunningham​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you....
Based on the instructions for creating an external table (see: https://docs.databricks.com/data-governance/unity-catalog/create-tables.html#create-a-table) I had assumed that external tables were a way to add an existing object store to Unity Catalog...
@Mark Miller​ :External tables in Databricks do not automatically receive external updates. When you create an external table in Databricks, you are essentially registering the metadata for an existing object store in Unity Catalog, which allows you ...
Does anyone know if its possible to hide the hive_metastore catalog from users or groups.
Hi @Jim Burris​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks!
I am setting up a new workspace that will use the Unity Catalog. I want all data stored in the Unity Catalog in the following catalogs: dev, staging, prod. I want to prevent users from accidentally reading and writing data elsewhere.For the above sit...
@Kevin Rossi​ @John Lourdu​ - I am also new to databricks setting up environment. Bu default "all users" have read access to below mentioned catalogs,my question is - i see an option to revoke read access, is it must have read access to all these ca...
I am trying to add a GCP storage as an external location and read and write Unity catalog enabled delta tables to that external location in GCP databricks. I keep getting the error that the databricks instance service principal doesn't have access to...
Hi there,I'm currently migrating Databricks (metastores, workspaces, etc.) from Azure to AWS using the Databricks migration tool. During the migration process and after digging in the code, I've come to the conclusion that the tool only supports a mi...
@Nino Weingart​ :Q1: You are correct that the Databricks migration tool only supports migration of the built-in Hive metastore and not custom metastores in Unity Catalog. If you want to migrate custom metastores, you would need to extend the code and...
Hello, there are a bunch of pages in the documentation that mention tables and views can be created via Terraform:> You can also create a managed table by using the Databricks Terraform provider and databricks_table. But those links to `databricks_ta...
Hmm it seems that Databricks developers say that creating tables/views in Unity Catalog from TerraForm is discouraged:> so there are quite a few gaps/edge cases with the tables API, hence customers should not use the API or Terraform to create/manage...
Hi Team, Where is the unity catalog placed? I know that it's attached to workspace. Is it some kind of VM? Where is it hosted?I'm able to access Unity catalog even though the clusters are down.BR,Rajath
Hi @Rajath C​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers your...
Best Practices for Securing Access to Data in DatabricksUnity Catalog is the unified governance solution for Data & AI assets in Databricks and greatly simplifies and centralized data access control. This guide includes best practices for both the st...
Let us know if this walkthrough helped you set up data access control and let us know how your journey to leveraging Unity Catalog is going!
We are planning on migrating to unity catalog but are unable to determine how we can segregate dev, staging and production data from each otherOur plan was to separate catalogs by SLDC Environment scopes (as per description and diagram at https://doc...
@Alex Davies​ :Unity Catalog does not currently support separating data by workspace or Azure subscription. As you noted, data from all catalogs within a region can be accessed by any workspace within that region, and it is up to user permissions to ...
I have managed to create my catalog and schema on unity catalog. Added Storage Blob Contributor role to the unity catalog metastore in ADLS for the Access Connector for Azure Databricks.however, when trying to create a table i received this error:Err...
Hi @adam wong​ Thank you for your question! To assist you better, please take a moment to review the answer and let me know if it best fits your needs.Please help us select the best solution by clicking on "Select As Best" if it does.Your feedback wi...
I click on the Search box on the top bar of the databricks workspace and select "Open advanced search". A dialog pops up with the Tables tab selected and an error below that says: "An error occurred while fetching search results. Please try again lat...
Hi @John Urban​ Hope everything is going great.Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tell us so we...
I have enabled Unity Catalog in my AWS Databricks workspace, switching from the standard Hive Metastore. I have a few custom Hive UDFs in a JAR file that I have copied to /databricks/jars/ folder through init script. I want to be able to register tho...
Hi @Franklin George​ Hope everything is going great.Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tell us ...
I'm using Azure Databricks.I've been following this guide to create a Unity Catalog metastore using a managed identity.When I create the catalog and the validation step kicks off, I get this screen: What does this mean?If I click "Force create" and t...
Hi @Alex Loddengaard​ Hope everything is going great.Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tell us...
Hi people, i`m using databricks Runtime 11.2 - Clusters Standard_F4 with Unity Catalog enabled and the service principals working,I literally perform an select in two delta tables that is in the storage X, the display() works, if i try to write an ne...
Hi @Guilherme Cesar​ Hope everything is going great.Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tell us ...
