Here's your Data + AI Summit 2024 - Data Governance recap as you navigate the explosion of AI, data and tools in efforts to build a flexible and scalable governance framework that spans your entire data and AI estate. Keynote: Evolving Data Governan...
Error in SQL statement: AnalysisException: SHOW PARTITIONS is not supported in Unity Catalog;What happened with the SHOW PARTITIONS command?Is this information available anywhere in the documentation?If I go to Databricks SQL documentation there is n...
I got answer from the support that there is no 'SHOW PARTITIONS' in Unity Catalog as "UC doesn't manage the partitions for tables".They suggested for work around `LIST` command, this is what I was able to came up with.The problem I see here, is for t...
Hi,Is it possible to create groups at the account level in Unity Catalog as a Service Principal ?I can manage to create groups when authenticated as a user, but not as a Service Principal. I then get an error "user not authorized". The service princi...
I have a few catalogs in databricks (using Unity Catalog). Each catalog (DEV, TEST, PROD) having its own external location (ADLSv2) and being tied to its own dbx workspace. Seeing how unity unifies the data, access policies and the metastore I would ...
Hi @Oscar Dyremyhr​ , looking at the error it looks like it is not able to resolve the hostname, could you please try to nslookup the hostname and see if it is getting resolved? Also, what is used in your connection string? ("account" can be used)
Anyone knows when will Unity Catalog be available on GCP?
@Prabakar Ammeappin​ @Debayan Mukherjee​ Any update on Unity Catalog preview for GCP please
Hi All, does anyone know the best way to provided feedback to the databricks team in regards to feature request or enhancements?This is a general question but I have immediate items I'm wishing to have improved: In the new data explorer for unity cat...
Hi @Ashley Betts​ You can open a feature request (Aha Idea) at https://databricks.aha.io/ Ideas registered here are reviewed by our product management team and will be picked up for implementation in the roadmap.The ideas here are generally listed fo...
Is there anyway to do this? From looking through the docs it seems like it can be set programmatically via CLI/Terraform but requires the assignment to removed first.The reason I need this is because the default is currently hive_metastore and when t...
@Rob Myers​ please login to SQL --> SQL Admin Console --> SQL Warehouse Settings (SQL Config parameters and try ) , if not use any warehouse and open sql editor and set below query %sqlSET spark.databricks.sql.initial.catalog.name = ...
i created unity catalog using terraform on AWS, and able to create manualy a meta store with same user.when trying to create metsastore from terraform, i get unauthorized error:Error: cannot create metastore: Unauthorized│ │ with module.unity_catalo...
I am new to Databricks and currently trying to setup Unity Catalog on AWS at Enterprise level. I would like to know if Unity Catalog is enterprise ready and if there are any pointers that could help me set is up on AWS at enterprise level.Appreciate ...
Hi @Yashpal Shah​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thank...
I am migrating my Data Lake to use Unity Catalog. However, this comes with changes to the clusters. I have tried a few options, but it seems rather complex than it should be. I need to create a cluster used by ADF that is Unity Enabled that can insta...
I exclusively use job clusters. They are cheaper.Especially when you create a pool with spot instances. I'd go for that because that is what it's made for: batch jobs/
Hi,I'm working with unity catalog for the last week. I'm refering to delta table by path, as follwing: path='s3://<my_bucket_name>/silver/data/<table_name>DeltaTable.forPath(spark, path)I get an exception that "is not a Delta table"using the table na...
its even more weired. on one next cells it doesnt... see below older version even by name doesnt work
I want to import data using the autoloader from a S3 bucket into a table which is managed inside a Unity Catalog.Right now, I run the code on an interactive cluster inside a notebook. In the future the code should run in a job cluster. The error I ge...
Hello,We're trying to use DBR 10.4 LTS in Shared Mode. But our python code is not working in that cluster.We're running into following messageYour administrator has only allowed sql commands on this cluster. This execution contained at least one disa...
Hi Parthsalvi,Could you please try using 11.1+ and see if that works?
I am trying to execute a UDF written in Python and receive the following error message:Error message: [UC_COMMAND_NOT_SUPPORTED] UDF/UDAF functions are not supported in Unity Catalog.;The limitation is not mentioned in the docs( https://docs.databric...
Indeed it is not mentioned in the limitations.This blog mentions python UDFs in UC, but in databricks SQL, not the classic DE workspace.So it seems that it is not (yet) supported, as the docs only mention SQL functions.
Do I have to transfer my data into Databricks' AWS account?,Candatabricks access my data and see all my compnay information
No all the data is stored within your own AWS.Some data to enhance your databricks experience maybe stored in DBFS
Hi,​Is there a know limitation in Unity Catalog workspaces ​with reading multiline json files from an S3 external location using pyspark? The file works perfectly fine if it's on dbfs, and reading json files in general works from that s3 location if ...
