Here's your Data + AI Summit 2024 - Data Governance recap as you navigate the explosion of AI, data and tools in efforts to build a flexible and scalable governance framework that spans your entire data and AI estate. Keynote: Evolving Data Governan...
Hey guys,I am trying to create a delta live table in Unity Catalog as follows:CREATE OR REFRESH STREAMING LIVE TABLE <catalog>.<db>.<table_name> AS SELECT ... However, I get the error: org.apache.spark.sql.AnalysisException: Unsupported SQL statemen...
@Anant Pingle​ As of right now, DLT doesn't support Unity Catalog yet. We can expect this feature in upcomig days.
HiI have just created a metastore to allow unity catalog to be enabled for my workspace. However I am getting the following error when trying to create a table as per the unity catalog demo notbook:Error in SQL statement: UnauthorizedAccessException:...
Thanks for your answer to the second part of my question, super helpful! Can you confirm the first part of my question as well please?
Hello, I am trying to test the new federation for Snowflake feature in Databricks SQL, following the docs (https://docs.databricks.com/query-federation/snowflake.html).Environment:Subscription: PremiumCompute: DB SQL (serverless)Metastore: Unity Cata...
Hi @Landan George​, thanks for the quick response. I get the following error if I try to run the SQL code from a Notebook:SQLException: No suitable driver found for jdbc:snowflake://app.snowflake.com/region/foo/Any suggestions?
I could able to save data using pyspark into S3 but not sure on how to save a file stream object into S3 bucket using pyspark. I could achieve this with help of python but when Unity catalog was enabled on Databrciks it always ends up with an access ...
I got to know that there is a change required at Unity-catalog to make it work with Python and got a recommendation to use pyspark to store file into S3.I do not see much information about storing a file stream object in an S3 bucket anywhere. Can an...
How can we connect to unity catalog tables to create REST API's using Python/.NET?
Hi @Sandhya SR​, We haven’t heard from you since the last response from @Jensen Ackles​ and I was checking to see if their suggestions helped you.Or else, If you have any solution, please share it with the community, as it can be helpful to others.Al...
I tried to create a view in hive_metastore.default which would access a table from a different catalog. Is there any chance to do so?eg.create view myTest as select * from someCatalog.someSchema.someTable
I will suggest the below links will help you, For exposing your Hive_metastore multiple catalogs, 1) Create 3 catalogs for each environment under single metastore by using Unity Catalog2) Now expose the Unity Catalog using DELTA SHARING to your BI ap...
We are setting up Databricks on our AWS account using customer managed VPC and enabling private link. I could see the private endpoints for frontend and backend with which we could spin up our EC2 instances. But the problem we are facing is to connec...
Este triángulo amoroso se puso complicado... ¡Faltan solo dÃas para el estreno de la segunda temporada de "Wildrose" en flixlatino!
Can we use existing AWS Glue catalog with Unity Catalog?
Hi @Rahul Mishra​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thank...
I have discovered some strange behavior when adding new columns to a table associated with Unity Catalog. I overwrote the data specifying overwriteSchema = True and passing in the abfss path as it is an external table. When quering against it, it is ...
Hi @Oscar Dyremyhr​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Tha...
Hello.I am using Azure Databricks since a few months and I want to add a Unity Catalog.Unfortunately, I have some problems:1. I am not able to create a Unity Catalog Cluster with settings:- Policy: Unrestricted- Multi Node with Single user access mod...
I'm facing the same issues as other customers listed above. I do not have the "Create Metastore" button available.
In setting up Databricks accounnt, we need to provide storage (s3 buckets) to store the final cleaned data or aggregated data. If the Unity Catalog is enabled at the beginning of account set up, we'd need to to provide S3 buckets again to store the m...
Hi @Tram Nguyen​, We haven’t heard from you since the last response from @Pat Sienkiewicz​ and @karthik p​​, and I was checking back to see if their suggestions helped you.Or else, If you have any solution, please share it with the community, as it c...
Hi,Does the Unity catalog support the Delta Live Tables? If yes, please help me with the documentation link. If not, any update on the feature readiness timeline would help. Thank you!
Hi @Venkadeshwaran K​, We haven’t heard from you since the last response from @Vivian Wilfred​​, and I was checking back to see if his suggestions helped you. Or else, If you have any solution, please share it with the community, as it can be helpful...
Hi Tried to create a delta table from spark data frame using below command:destination_path = "/dbfs/mnt/kidneycaredevstore/delta/df_corr_feats_spark_4"df_corr_feats_spark.write.format("delta").option("delta.columnMapping.mode", "name").option("path"...
Hi @Shafiul Alam​ ,who gave those names to columns? you can rename you columns, replace spaces / special characters, for example:%python import re list_of_columns = df_corr_feats_spark.colums renamed_list_of_columns = [ re.sub(r'[^0-9a-zA-Z]+', "_", ...
Hi all, I have set up a Unity Catalog (UC), and I am attempting to "upgrade my Delta tables" in the hive_metastore to my new catalog.I have followed the instructions here: https://learn.microsoft.com/en-us/azure/databricks/data-governance/unity-catal...
FYI - this is now magically resolved. I'm guessing it was a temporary bug or something.​
