I have discovered some strange behavior when adding new columns to a table associated with Unity Catalog. I overwrote the data specifying overwriteSchema = True and passing in the abfss path as it is an external table. When quering against it, it is ...
Hi @Oscar Dyremyhr​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Tha...
Hello.I am using Azure Databricks since a few months and I want to add a Unity Catalog.Unfortunately, I have some problems:1. I am not able to create a Unity Catalog Cluster with settings:- Policy: Unrestricted- Multi Node with Single user access mod...
In setting up Databricks accounnt, we need to provide storage (s3 buckets) to store the final cleaned data or aggregated data. If the Unity Catalog is enabled at the beginning of account set up, we'd need to to provide S3 buckets again to store the m...
Hi @Tram Nguyen​, We haven’t heard from you since the last response from @Pat Sienkiewicz​ and @karthik p​​, and I was checking back to see if their suggestions helped you.Or else, If you have any solution, please share it with the community, as it c...
Hi,Does the Unity catalog support the Delta Live Tables? If yes, please help me with the documentation link. If not, any update on the feature readiness timeline would help. Thank you!
Hi @Venkadeshwaran K​, We haven’t heard from you since the last response from @Vivian Wilfred​​, and I was checking back to see if his suggestions helped you. Or else, If you have any solution, please share it with the community, as it can be helpful...
Hi Tried to create a delta table from spark data frame using below command:destination_path = "/dbfs/mnt/kidneycaredevstore/delta/df_corr_feats_spark_4"df_corr_feats_spark.write.format("delta").option("delta.columnMapping.mode", "name").option("path"...
Hi @Shafiul Alam​ ,who gave those names to columns? you can rename you columns, replace spaces / special characters, for example:%python
import re
list_of_columns = df_corr_feats_spark.colums
renamed_list_of_columns = [ re.sub(r'[^0-9a-zA-Z]+', "_", ...
Hi all, I have set up a Unity Catalog (UC), and I am attempting to "upgrade my Delta tables" in the hive_metastore to my new catalog.I have followed the instructions here: https://learn.microsoft.com/en-us/azure/databricks/data-governance/unity-catal...
@Brinda Gupta​ I found this link: Unity Catalog - DatabricksHere it is mentioned you can use "ALTER TABLE <TABLENAME> ALTER COLUMN <COLUMNNAME> SET MASK <MASKNAME>I tried this but it says there is no keyword like MASK.Please help me to understand thi...
Hi, We have some problems to create some resources using terraform, after the Unity catalog migration. We have created a group and SVC Principal under account as in the doc using terraform. (AWS infrastructure) (https://registry.terraform.io/provider...
Hi @Jordi Casanella​ ,I have been working with terraform for databricks lately and I would say that I had to switch my approach couple of times due to issues like you have right now (account vs workspace level API).I assume that with this part you di...
Error in SQL statement: AnalysisException: SHOW PARTITIONS is not supported in Unity Catalog;What happened with the SHOW PARTITIONS command?Is this information available anywhere in the documentation?If I go to Databricks SQL documentation there is n...
I got answer from the support that there is no 'SHOW PARTITIONS' in Unity Catalog as "UC doesn't manage the partitions for tables".They suggested for work around `LIST` command, this is what I was able to came up with.The problem I see here, is for t...
Hi,Is it possible to create groups at the account level in Unity Catalog as a Service Principal ?I can manage to create groups when authenticated as a user, but not as a Service Principal. I then get an error "user not authorized". The service princi...
I have a few catalogs in databricks (using Unity Catalog). Each catalog (DEV, TEST, PROD) having its own external location (ADLSv2) and being tied to its own dbx workspace. Seeing how unity unifies the data, access policies and the metastore I would ...
Hi @Oscar Dyremyhr​ , looking at the error it looks like it is not able to resolve the hostname, could you please try to nslookup the hostname and see if it is getting resolved? Also, what is used in your connection string? ("account" can be used)
Hi All, does anyone know the best way to provided feedback to the databricks team in regards to feature request or enhancements?This is a general question but I have immediate items I'm wishing to have improved: In the new data explorer for unity cat...
Hi @Ashley Betts​ You can open a feature request (Aha Idea) at https://databricks.aha.io/ Ideas registered here are reviewed by our product management team and will be picked up for implementation in the roadmap.The ideas here are generally listed fo...
Is there anyway to do this? From looking through the docs it seems like it can be set programmatically via CLI/Terraform but requires the assignment to removed first.The reason I need this is because the default is currently hive_metastore and when t...
@Rob Myers​ please login to SQL --> SQL Admin Console --> SQL Warehouse Settings (SQL Config parameters and try ) , if not use any warehouse and open sql editor and set below query %sqlSET spark.databricks.sql.initial.catalog.name = ...
i created unity catalog using terraform on AWS, and able to create manualy a meta store with same user.when trying to create metsastore from terraform, i get unauthorized error:Error: cannot create metastore: Unauthorized│ │ with module.unity_catalo...
