Here's your Data + AI Summit 2024 - Data Governance recap as you navigate the explosion of AI, data and tools in efforts to build a flexible and scalable governance framework that spans your entire data and AI estate. Keynote: Evolving Data Governan...
What best practices Can I find the DB governance solution to control costs across different cloud providers an on prem environments?
Hello,Will Unity catalog work with a mix of streaming and batch tables?
Hi @BankBricks,Unity Catalog supports structured streaming as well:https://docs.databricks.com/structured-streaming/unity-catalog.htmlFundamentally you have the same availability of features as with your standard hive structure within databricks - bu...
Day 2 of the DAIS 2023: Intro to Untity Catalog and how to use it to ensure data governance on a multi region and even multi cloud level!
Having a great start to the Data+AI Summit 2023. Especially interested to deep dive into the Unity Catalog and Data governance sessions. Excited at the prospect of Databricks providing a one stop shop for all Data engineering and Data Science require...
Hi ,I am trying to get list of all the metastores associated with an accountId. For this I am using the below REST API to get data:accountId = json.loads(dbutils.notebook.entry_point.getDbutils().notebook().getContext().toJson())["tags"]["accountId"]...
Apparently, it is working if I add the header `X-Databricks-Account-Console-API-Version` with value `2.0` to the call
Hi thereI have used my company email to register an account for customer-academy.databricks.com a while back.Now what I need to do is create an account with partner-academy.databricks.com using my company email too.However when I register at partner-...
Hi @Samidha Sahu​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers ...
I have a problem with upgrading table to Unity Catalog, I got the error:summary:Error in SQL statement:AnalysisException:org.apache.hadoop.hive.ql.metadata.HiveException:Unable to fetch table calexception. Exception thrown when executing query :SELEC...
Hi guys, new to we have a unity catalog setup already supporting 4 of our workspaces and we would like to rename the metastore. What I would like to confirm is whether or not would I require any configurational or code changes after renaming?
Hi @Syed Ismail​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers y...
Hello,Im using Unity Catalog managed tables and Unity Catalog DLTs. I urgently need to use a UDF in DBSQL because pySpark UDFs dont work in UC. I found this way to create a SQL function and use python language inside:CREATE FUNCTION redact(a STRING) ...
It will be available soon in public preview. You will need to have a Pro SQL Warehouse or a Serverless SQL Warehouse to use Python UDF in Databricks SQL.
Databricks-connect 11.3 works for unity catalog enabled cluster or not ? I am trying to utilize the databricks-connect in Azure DevOps pipeline using Microsoft hosted agent. But getting the below error when running the databricks-connect test command...
Hi @Rey Jhon​ Sadly, databricks-connect for DBR versions < 13 does not support UC, as outlined in the limitations section: https://docs.databricks.com/dev-tools/databricks-connect-legacy.html#limitations.To use UC, we recommend using DBR version >= 1...
We're planning on adding additional workspaces in different Azure resource groups to separate lines of business with the goal of Unity Catalog spanning these workspaces. Each Resource Group will contain a workspace and a data lake. When it comes to U...
We've done similar on AWS, If this is regarding setting up backend storage for the metastore it can have a separate one (Ideally it should). Each catalog for each business unit/ workspace can have its own backend data lake too. This one would overrid...
Using Unity Catalog as a unified metastore for Databricks we are able to track the data lineage of tables. The lineage is going to be maintained for 30 days - this is described in the official documentation: - Because lineage is computed on a 30-day ...
In databricks course V3 Unity catalog session I could not understand the answer of this question, can someone please enlighten me ? A senior data engineer with workspace identity privileges has created the schema customers within a Unity Catalog name...
The anwser for this question is mentionned here https://community.databricks.com/s/question/0D58Y0000ACa06GSQR/is-there-any-hierarchy-within-schema-permissions
In Unity catalog part we have this question: A data engineer needs to copy an external table from their default Hive metastore to the Unity Catalog metastore. Which of the following are required to upgrade the table to be managed by Unity Catalog? Se...
I think it was a mistake in the mentionned correction in Databricks according to the documentation Upgrade a single external table to Unity CatalogYou can copy an external table from your default Hive metastore to the Unity Catalog metastore using th...
Have been informed by a vendor we are using for Data Observability that the Unity Catalog lineage can be accessed via the systems schema. This is not enabled by default and we will need to execute a PUT request to the endpoint below. I am unable to f...
Hi @Stephen Quarshie​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.T...
