Here's your Data + AI Summit 2024 - Data Governance recap as you navigate the explosion of AI, data and tools in efforts to build a flexible and scalable governance framework that spans your entire data and AI estate. Keynote: Evolving Data Governan...
HiI have just created a metastore to allow unity catalog to be enabled for my workspace. However I am getting the following error when trying to create a table as per the unity catalog demo notbook:Error in SQL statement: UnauthorizedAccessException:...
Thanks for your answer to the second part of my question, super helpful! Can you confirm the first part of my question as well please?
Hello, I am trying to test the new federation for Snowflake feature in Databricks SQL, following the docs (https://docs.databricks.com/query-federation/snowflake.html).Environment:Subscription: PremiumCompute: DB SQL (serverless)Metastore: Unity Cata...
Hi @Landan George​, thanks for the quick response. I get the following error if I try to run the SQL code from a Notebook:SQLException: No suitable driver found for jdbc:snowflake://app.snowflake.com/region/foo/Any suggestions?
I could able to save data using pyspark into S3 but not sure on how to save a file stream object into S3 bucket using pyspark. I could achieve this with help of python but when Unity catalog was enabled on Databrciks it always ends up with an access ...
I got to know that there is a change required at Unity-catalog to make it work with Python and got a recommendation to use pyspark to store file into S3.I do not see much information about storing a file stream object in an S3 bucket anywhere. Can an...
How can we connect to unity catalog tables to create REST API's using Python/.NET?
We are setting up Databricks on our AWS account using customer managed VPC and enabling private link. I could see the private endpoints for frontend and backend with which we could spin up our EC2 instances. But the problem we are facing is to connec...
Este triángulo amoroso se puso complicado... ¡Faltan solo dÃas para el estreno de la segunda temporada de "Wildrose" en flixlatino!
Can we use existing AWS Glue catalog with Unity Catalog?
Hi @Rahul Mishra​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thank...
I have discovered some strange behavior when adding new columns to a table associated with Unity Catalog. I overwrote the data specifying overwriteSchema = True and passing in the abfss path as it is an external table. When quering against it, it is ...
Hi @Oscar Dyremyhr​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Tha...
Hello.I am using Azure Databricks since a few months and I want to add a Unity Catalog.Unfortunately, I have some problems:1. I am not able to create a Unity Catalog Cluster with settings:- Policy: Unrestricted- Multi Node with Single user access mod...
I'm facing the same issues as other customers listed above. I do not have the "Create Metastore" button available.
Hi Tried to create a delta table from spark data frame using below command:destination_path = "/dbfs/mnt/kidneycaredevstore/delta/df_corr_feats_spark_4"df_corr_feats_spark.write.format("delta").option("delta.columnMapping.mode", "name").option("path"...
Hi @Shafiul Alam​ ,who gave those names to columns? you can rename you columns, replace spaces / special characters, for example:%python import re list_of_columns = df_corr_feats_spark.colums renamed_list_of_columns = [ re.sub(r'[^0-9a-zA-Z]+', "_", ...
Hi all, I have set up a Unity Catalog (UC), and I am attempting to "upgrade my Delta tables" in the hive_metastore to my new catalog.I have followed the instructions here: https://learn.microsoft.com/en-us/azure/databricks/data-governance/unity-catal...
FYI - this is now magically resolved. I'm guessing it was a temporary bug or something.​
In setting up Databricks accounnt, we need to provide storage (s3 buckets) to store the final cleaned data or aggregated data. If the Unity Catalog is enabled at the beginning of account set up, we'd need to to provide S3 buckets again to store the m...
@Tram Nguyen​ above pat provided links will have clear information. but to provide overview account s3 bucket should be different unity catalog s3 bucket should be different (Meta store in technical terms)unity catalog bucket should be configured in ...
Hi,Does the Unity catalog support the Delta Live Tables? If yes, please help me with the documentation link. If not, any update on the feature readiness timeline would help. Thank you!
Hi @Venkadeshwaran K​ ,No, UC does not support DLT as of today. We have this feature on the timeline. We will be having a private preview by end of this year for DLT and UC integration for SQL. There is no ETA yet for GA but you can definitely expect...
@Brinda Gupta​ I found this link: Unity Catalog - DatabricksHere it is mentioned you can use "ALTER TABLE <TABLENAME> ALTER COLUMN <COLUMNNAME> SET MASK <MASKNAME>I tried this but it says there is no keyword like MASK.Please help me to understand thi...
Hi, We have some problems to create some resources using terraform, after the Unity catalog migration. We have created a group and SVC Principal under account as in the doc using terraform. (AWS infrastructure) (https://registry.terraform.io/provider...
Hi @Jordi Casanella​ ,I have been working with terraform for databricks lately and I would say that I had to switch my approach couple of times due to issues like you have right now (account vs workspace level API).I assume that with this part you di...
