Here's your Data + AI Summit 2024 - Data Governance recap as you navigate the explosion of AI, data and tools in efforts to build a flexible and scalable governance framework that spans your entire data and AI estate. Keynote: Evolving Data Governan...
Hi,I have two questions about cluster logs.Q1) It seems that I can only store logs on DBFS. But, is there any way I can send the cluster logs to an Azure storage account?Note that workspace is not enabled for the Unity Catalog (and is not aimed to be...
Hi @Mohammad Saber Hope everything is going great.Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tell us s...
Hello,Is it possible to use multiple Hive catalogs with Databricks? I know it can be done with Unity Catalog, but we're the only workspace admins in our organization. As far as I understand, we'd need account-level access to set up Unity Catalog.Best...
Hi @Chris Nawara Hope everything is going great.Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tell us so ...
Hi there,We have Unity Catalogs enabled in Azure Databricks.I would like to connect to system.information_schema.* or any other *.information_schema.*. Querying in UI or other IDEs works as expected - select * from system.information_schema.tables.No...
Hi @Egor Larin Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks!
Hello, I have been trying to solidify my understanding of what changes occur after I upgrade my data to Unity Catalog. Our current ingestion process incrementally loads new data to tables in the hive metastore which are mounted to an s3 bucket. If I ...
Hi @Rachel Cunningham Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you....
Based on the instructions for creating an external table (see: https://docs.databricks.com/data-governance/unity-catalog/create-tables.html#create-a-table) I had assumed that external tables were a way to add an existing object store to Unity Catalog...
@Mark Miller :External tables in Databricks do not automatically receive external updates. When you create an external table in Databricks, you are essentially registering the metadata for an existing object store in Unity Catalog, which allows you ...
Does anyone know if its possible to hide the hive_metastore catalog from users or groups.
Hi @Jim Burris Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks!
I am setting up a new workspace that will use the Unity Catalog. I want all data stored in the Unity Catalog in the following catalogs: dev, staging, prod. I want to prevent users from accidentally reading and writing data elsewhere.For the above sit...
@Kevin Rossi @John Lourdu - I am also new to databricks setting up environment. Bu default "all users" have read access to below mentioned catalogs,my question is - i see an option to revoke read access, is it must have read access to all these ca...
I am trying to add a GCP storage as an external location and read and write Unity catalog enabled delta tables to that external location in GCP databricks. I keep getting the error that the databricks instance service principal doesn't have access to...
Hi there,I'm currently migrating Databricks (metastores, workspaces, etc.) from Azure to AWS using the Databricks migration tool. During the migration process and after digging in the code, I've come to the conclusion that the tool only supports a mi...
@Nino Weingart :Q1: You are correct that the Databricks migration tool only supports migration of the built-in Hive metastore and not custom metastores in Unity Catalog. If you want to migrate custom metastores, you would need to extend the code and...
Hello, there are a bunch of pages in the documentation that mention tables and views can be created via Terraform:> You can also create a managed table by using the Databricks Terraform provider and databricks_table. But those links to `databricks_ta...
Hmm it seems that Databricks developers say that creating tables/views in Unity Catalog from TerraForm is discouraged:> so there are quite a few gaps/edge cases with the tables API, hence customers should not use the API or Terraform to create/manage...
Hi Team, Where is the unity catalog placed? I know that it's attached to workspace. Is it some kind of VM? Where is it hosted?I'm able to access Unity catalog even though the clusters are down.BR,Rajath
Hi @Rajath C Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers your...
Best Practices for Securing Access to Data in DatabricksUnity Catalog is the unified governance solution for Data & AI assets in Databricks and greatly simplifies and centralized data access control. This guide includes best practices for both the st...
Let us know if this walkthrough helped you set up data access control and let us know how your journey to leveraging Unity Catalog is going!
We are planning on migrating to unity catalog but are unable to determine how we can segregate dev, staging and production data from each otherOur plan was to separate catalogs by SLDC Environment scopes (as per description and diagram at https://doc...
@Alex Davies :Unity Catalog does not currently support separating data by workspace or Azure subscription. As you noted, data from all catalogs within a region can be accessed by any workspace within that region, and it is up to user permissions to ...
I have managed to create my catalog and schema on unity catalog. Added Storage Blob Contributor role to the unity catalog metastore in ADLS for the Access Connector for Azure Databricks.however, when trying to create a table i received this error:Err...
Hi @adam wong Thank you for your question! To assist you better, please take a moment to review the answer and let me know if it best fits your needs.Please help us select the best solution by clicking on "Select As Best" if it does.Your feedback wi...
