Here's your Data + AI Summit 2024 - Data Governance recap as you navigate the explosion of AI, data and tools in efforts to build a flexible and scalable governance framework that spans your entire data and AI estate. Keynote: Evolving Data Governan...
Can we use existing AWS Glue catalog with Unity Catalog?
Hi @Rahul Mishra​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thank...
I have discovered some strange behavior when adding new columns to a table associated with Unity Catalog. I overwrote the data specifying overwriteSchema = True and passing in the abfss path as it is an external table. When quering against it, it is ...
Hi @Oscar Dyremyhr​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Tha...
Hello.I am using Azure Databricks since a few months and I want to add a Unity Catalog.Unfortunately, I have some problems:1. I am not able to create a Unity Catalog Cluster with settings:- Policy: Unrestricted- Multi Node with Single user access mod...
I'm facing the same issues as other customers listed above. I do not have the "Create Metastore" button available.
Hi Tried to create a delta table from spark data frame using below command:destination_path = "/dbfs/mnt/kidneycaredevstore/delta/df_corr_feats_spark_4"df_corr_feats_spark.write.format("delta").option("delta.columnMapping.mode", "name").option("path"...
Hi @Shafiul Alam​ ,who gave those names to columns? you can rename you columns, replace spaces / special characters, for example:%python import re list_of_columns = df_corr_feats_spark.colums renamed_list_of_columns = [ re.sub(r'[^0-9a-zA-Z]+', "_", ...
Hi all, I have set up a Unity Catalog (UC), and I am attempting to "upgrade my Delta tables" in the hive_metastore to my new catalog.I have followed the instructions here: https://learn.microsoft.com/en-us/azure/databricks/data-governance/unity-catal...
FYI - this is now magically resolved. I'm guessing it was a temporary bug or something.​
In setting up Databricks accounnt, we need to provide storage (s3 buckets) to store the final cleaned data or aggregated data. If the Unity Catalog is enabled at the beginning of account set up, we'd need to to provide S3 buckets again to store the m...
@Tram Nguyen​ above pat provided links will have clear information. but to provide overview account s3 bucket should be different unity catalog s3 bucket should be different (Meta store in technical terms)unity catalog bucket should be configured in ...
Hi,Does the Unity catalog support the Delta Live Tables? If yes, please help me with the documentation link. If not, any update on the feature readiness timeline would help. Thank you!
Hi @Venkadeshwaran K​ ,No, UC does not support DLT as of today. We have this feature on the timeline. We will be having a private preview by end of this year for DLT and UC integration for SQL. There is no ETA yet for GA but you can definitely expect...
@Brinda Gupta​ I found this link: Unity Catalog - DatabricksHere it is mentioned you can use "ALTER TABLE <TABLENAME> ALTER COLUMN <COLUMNNAME> SET MASK <MASKNAME>I tried this but it says there is no keyword like MASK.Please help me to understand thi...
Hi, We have some problems to create some resources using terraform, after the Unity catalog migration. We have created a group and SVC Principal under account as in the doc using terraform. (AWS infrastructure) (https://registry.terraform.io/provider...
Hi @Jordi Casanella​ ,I have been working with terraform for databricks lately and I would say that I had to switch my approach couple of times due to issues like you have right now (account vs workspace level API).I assume that with this part you di...
Error in SQL statement: AnalysisException: SHOW PARTITIONS is not supported in Unity Catalog;What happened with the SHOW PARTITIONS command?Is this information available anywhere in the documentation?If I go to Databricks SQL documentation there is n...
I got answer from the support that there is no 'SHOW PARTITIONS' in Unity Catalog as "UC doesn't manage the partitions for tables".They suggested for work around `LIST` command, this is what I was able to came up with.The problem I see here, is for t...
Hi,Is it possible to create groups at the account level in Unity Catalog as a Service Principal ?I can manage to create groups when authenticated as a user, but not as a Service Principal. I then get an error "user not authorized". The service princi...
I have a few catalogs in databricks (using Unity Catalog). Each catalog (DEV, TEST, PROD) having its own external location (ADLSv2) and being tied to its own dbx workspace. Seeing how unity unifies the data, access policies and the metastore I would ...
Hi @Oscar Dyremyhr​ , looking at the error it looks like it is not able to resolve the hostname, could you please try to nslookup the hostname and see if it is getting resolved? Also, what is used in your connection string? ("account" can be used)
Anyone knows when will Unity Catalog be available on GCP?
@Prabakar Ammeappin​ @Debayan Mukherjee​ Any update on Unity Catalog preview for GCP please
Hi All, does anyone know the best way to provided feedback to the databricks team in regards to feature request or enhancements?This is a general question but I have immediate items I'm wishing to have improved: In the new data explorer for unity cat...
Hi @Ashley Betts​ You can open a feature request (Aha Idea) at https://databricks.aha.io/ Ideas registered here are reviewed by our product management team and will be picked up for implementation in the roadmap.The ideas here are generally listed fo...
Is there anyway to do this? From looking through the docs it seems like it can be set programmatically via CLI/Terraform but requires the assignment to removed first.The reason I need this is because the default is currently hive_metastore and when t...
@Rob Myers​ please login to SQL --> SQL Admin Console --> SQL Warehouse Settings (SQL Config parameters and try ) , if not use any warehouse and open sql editor and set below query %sqlSET spark.databricks.sql.initial.catalog.name = ...
