Here's your Data + AI Summit 2024 - Data Governance recap as you navigate the explosion of AI, data and tools in efforts to build a flexible and scalable governance framework that spans your entire data and AI estate. Keynote: Evolving Data Governan...
We have multiple processing chains that uses R notebooks with Sparklyr, and we try to migrate them from runtime 10.4 to 12.2. unfortunately, there seems to be an incompatibility with Sparlyr with runtimes > 10.4:Steps to reproduce:1) Create a noteboo...
Hi @Robin LOCHE​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers y...
Hi,I am trying to setup presto in aws emr to be able to read delta tables. For this, as per my understanding, I need to setup the delta.properties file. Does anybody know how do I find the hive.metastore.uri that is needed for this file? For either u...
To find the hive.metastore.uri needed for the delta.properties file, you can follow the below steps:Login to your AWS Glue consoleClick on the "Crawlers" tab and select the Glue crawler that you are using for your Delta tablesIn the "Crawler details"...
We are following these instructions to link/assign a unity catalog metastore to a workspace using the UI. When we apply the option to enable the unity metastore in the workspace configuration, after hitting "Update", the option gets disabled. When we...
@Guilherme Jordan​ :If the option to enable the Unity Catalog metastore in the workspace configuration is getting disabled after hitting "Update", it could be a sign of an issue with the workspace or the Unity Catalog metastore.Here are some steps yo...
I've been trying to set up a Unity Catalog and play around with it. For that purpose, I created a new workspace with premium tier. As a tutorial, I follow Dustin Vannoy's video (https://www.youtube.com/watch?v=-RwzDRVgjLc) where he goes to his userna...
Hi @Martin Kostadinov​ Thank you for your question! To assist you better, please take a moment to review the answer and let me know if it best fits your needs.Please help us select the best solution by clicking on "Select As Best" if it does.Your fee...
Hello. Despite having Databricks Premium, I cannot find the options to work with Unity Catalog. I login as administrator, go to the Data section, but cannot find the option to create a Unity Metastore. What am I missing?
Hi @Vlad Gheorghe​ Hope everything is going great.Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tell us so...
Apologies in advance for the soft question, but I'm genuinely struggling with this.We're a small data science unit just setting up in Databricks. While we do run some intensive ETL and analytics jobs, a non-trivial part of the team's BAU is explorato...
Hi @Jason Millburn​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answer...
If I will migrate the Hive_metastore table into Unity Catalog table and subsequently hive metastore table is target of curation process. how I will migrate the delta record from Hive_metastore table to Unity Catalog table
Hi @Vivek Singh Banafar​ Hope everything is going great.Just checking in. Does @Debayan Mukherjee​'s response answer your question? If yes, would you be happy to mark it as best so that other members can find the solution more quickly?We'd love to he...
After implementing Unity catalog, assuming a user has access to more than one DB. Can that user access all the DB in a single workspace? Can we restrict any specific DB in a workspace?
Hi @Prashant Lalaiya​,Hope you are well. Just wanted to see if you were able to find an answer to your question and would you like to mark an answer as best? It would be really helpful for the other members too.Cheers!
Hi all,I see a number of metastore down events in databricks workspace: do any one know what's the reason why this happens?Thanks,Barney
Hi @Ron Barneveld​ Could you kindly help with more details? Is this is a production metastore that you are mentioning or any metastore created by you for test environment. Also where it is hosted?
Hi,dbutils.fs.ls() is a most useful utility to list the files in remote cloud storage. I cannot use this function on top of the location managed by the Unity catalog.I have a managed external location created on `'abfss://xyz@abcd.dfs.core.windows.ne...
Hi @Venkadeshwaran K​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answ...
I have databricks premium and I want to enable unity catalog on it, I am not able to do so. I followed the following steps which are attached in the attachments. I am also attaching the screenshot of my cluster configuration. When I run this command:...
Hi @Prerna Munjal​ Help us build a vibrant and resourceful community by recognizing and highlighting insightful contributions. Mark the best answers and show your appreciation!Regards
Currently we have an Init script that specifies the location of different network resources by updating the hosts file. When we moved to the Unity catalog all of our clusters had to be upgraded to support it. Now our scripts have stopped working due ...
Is it best practice to separate unity catalog meta store ADLS Gen2 separate from ADLS Gen 2 to store data ?Since per region only one meta store can be created, will there be a separate meta store for PROD, and NON-PROD(QA and DEV)? If yes they need t...
@Ashok Zubrewar​ coming to your 3 rd question, if you are using any external tables then non uc ADLS GEN 2 is mandatory, you can not use UC ADLS GEN2. as it hosts metadata and managed table data. there is no restriction in terms of your external buc...
I was wondering if there is a best practice or recommended way to organize data objects (tables) in Unity Catalog.Following the medallion architecture, how should one organize and name these three namespaces:Catalog(s)Schema(s)Table(s)Example:Catalog...
In Real time do we have 1 meta store for PROD and NON-PROD ? if Yes in the backend there would be a one storage account (ADLS Gen2) > Container for all Environments.
Free e-book: Rise of the Data LakehouseExciting news! You can now get a free copy of Bill Inmon's new book, Rise of the Data Lakehouse. The book is a must-read for anyone looking to implement the data lakehouse approach in their organization. The dat...
