- 1968 Views
- 2 replies
- 1 kudos
Unity Catalog metastore list is showing empty
Hi ,I am not able to list the meta-stores in databricks cli using the below command :databricks unity-catalog metastores list{}but when I tried databricks unity-catalog metastores get-summary I am able to get the meta-store info .Can anyone help me ...
- 1968 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @Kaniz Fatma​ , Unity catalog is enabled in my workspace and i have been assigned metastore admin and account admin also.databricks unity-catalog metastores list --debugHTTP debugging enabledsend: b'GET /api/2.1/unity-catalog/metastores HTTP/1.1\r...
- 1 kudos
- 2919 Views
- 3 replies
- 6 kudos
Resolved! Has Unity Catalog been available in Azure Gov Cloud?
We are using Databricks with Premium Tier in Azure Gov Cloud. We check the Data section but don't see any options to Create Metastore.
- 2919 Views
- 3 replies
- 6 kudos
- 6 kudos
Azure.gov does not have Unity Catalog (as of July 2024). I think previous responses missed the context of government cloud in OP's question. UC has been open sourced since this question was asked, and is a more comprehensive solution in commercial cl...
- 6 kudos
- 6419 Views
- 12 replies
- 3 kudos
Resolved! Unity catalog - How do you modify groups properly ?
Hello,What is the best practice to modify/delete/recreate groups properly ?In order to rename a group, the only mean was to delete/recreate. But after deletion in the account console, the permissions granted to the deleted groups in the tables were i...
- 6419 Views
- 12 replies
- 3 kudos
- 3 kudos
Hello,I have exactly the same issue - I am also using terraform.I deleted a group and the catalog permissions are in bad state. I am not able to revoke access to this group using the Databricks UI nor REST API. I also tried to recreate the group wit...
- 3 kudos
- 10648 Views
- 10 replies
- 3 kudos
Shared cluster configuration that permits `dbutils.fs` commands
My workspace has a couple different types of clusters, and I'm having issues using the `dbutils` filesystem utilities when connected to a shared cluster. I'm hoping you can help me fix the configuration of the shared cluster so that I can actually us...
- 10648 Views
- 10 replies
- 3 kudos
- 3 kudos
Can you not use a No Isolation Shared cluster with Table access controls enabled on workspace level?
- 3 kudos
- 2450 Views
- 3 replies
- 0 kudos
Primary Key not working as expected on Unity Catalog delta tables
Hi @Chetan Kardekar. I noticed that you had commented on Primary key on Delta tables. Do we have that feature already released in DataBricks Premium. I have a Unity Catalog and I created a table with Primary Key, though it doesnt act like Primary Key...
- 2450 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi @Bala Akas​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks!
- 0 kudos
- 12022 Views
- 8 replies
- 3 kudos
UnauthorizedAccessException: PERMISSION_DENIED: User does not have READ FILES on External Location
Our jobs have been running fine so far w/o any issues on a specific workspace. These jobs read data from files on Azure ADLS storage containers and dont use the hive metastore data at all.Now we attached the unity metastore to this workspace, created...
- 12022 Views
- 8 replies
- 3 kudos
- 3 kudos
@Wojciech_BUK I granted both in the GUI:) you can either search for display name there (and then it uses the Managed Identity Object ID), or you can search directly for the value of the "Managed Identity Application ID" and then it works correctly! ...
- 3 kudos
- 6375 Views
- 3 replies
- 0 kudos
Unable to use unity catalog in notebook
com.databricks.backend.common.rpc.SparkDriverExceptions$SQLExecutionException: org.apache.spark.sql.connector.catalog.CatalogNotFoundException: Catalog 'uc-dev' plugin class not found: spark.sql.catalog.uc-dev is not defined ....I get the above when ...
- 6375 Views
- 3 replies
- 0 kudos
- 0 kudos
I had the same error plugin class not found: spark.sql.catalog... is not defined immediately after attaching the workspace into Unity catalog.The error was resolved by restarting SQL Warehouse.It seems that if SQL Warehouse (or any cluster) is runnin...
- 0 kudos
- 7758 Views
- 4 replies
- 1 kudos
Accidentally removing the service principal that owns the view seems to put the Unity Catalog in an illegal state. Can you fix this?
I renamed our service principal in Terraform, which forces a replacement where the old service principal is removed and a new principal with the same permission is recreated. The Terraform succeeds to apply, but when I try to run dbt that creates tab...
- 7758 Views
- 4 replies
- 1 kudos
- 1 kudos
This is also true for removing groups before unassigning them (removing and unassigning in Terraform)│ Error: cannot update grants: Could not find principal with name <My Group Name>
- 1 kudos
- 13713 Views
- 9 replies
- 11 kudos
input_file_name() not supported in Unity Catalog
Hey, so our notebooks reading a bunch of json files from storage typically use a input_file_name() when moving from raw to bronze, but after upgrading to Unity Catalog we get an error message:AnalysisException: [UC_COMMAND_NOT_SUPPORTED] input_file_n...
- 13713 Views
- 9 replies
- 11 kudos
- 11 kudos
.withColumn("RECORD_FILE_NAME", col("_metadata.file_name"))Will work for spark.read to get the file name, or: .withColumn("RECORD_FILE_NAME", col("_metadata.file_path"))To get the whole file path
- 11 kudos
- 6238 Views
- 6 replies
- 2 kudos
Resolved! How to grant access to views without granting access to underlying tables
I have a medallion architecture: Bronze layer: Raw data in tablesSilver layer: Refined data in views created from the bronze layerGold layer: Data products as views created from the silver layerCurrently I have a data scientist that needs access to d...
- 6238 Views
- 6 replies
- 2 kudos
- 2 kudos
Single-user clusters use a different security mode which is the reason for this difference. On single-user/assigned clusters, you'll need the Fine Grained Access Control service (which is a Serverless service) - that is the solution to this problem (...
- 2 kudos
- 9130 Views
- 4 replies
- 1 kudos
Connection to Azure SQL Server: ODBC Driver 18 for SQL Server
Task: Setup connection to Azure SQL Server.A couple things have changed...*We've started using Unity Catalog, so need Unity Catalog -enabled clusters*Legacy init scripts have been deprecated, and this is how we had our pyodbc setup, etc. defined.Code...
- 9130 Views
- 4 replies
- 1 kudos
- 1 kudos
Hi @js54123875 did you manage to find a solution for this? I'm facing a similar problem.ThanksDiego
- 1 kudos
- 5823 Views
- 3 replies
- 6 kudos
Bug? Unity Catalog incompatible with Sparklyr in RStudio (on Driver) and as well if used on one cluster from multiple notebooks?
If I start a RStudio Server with in cluster init script as described here in a Unity Catalog Cluster the sparklyr connection fails with an error about a missing Credential Scope.=LI tried it both in 11.3LTS and 12.0 Beta. I tried it only in a Persona...
- 5823 Views
- 3 replies
- 6 kudos
- 6 kudos
Have run into this issue as well. Let me know if there was any resolution
- 6 kudos
- 8459 Views
- 3 replies
- 3 kudos
Resolved! Error creating external location in Unity Catalog
HelloWhen I try to create an external location I get this error:Failed to access cloud storage: [AbfsRestOperationException] HTTP Error -1CustomTokenProvider getAccessToken threw com.databricks.api.base.DatabricksServiceException : INTERNAL_ERROR: Un...
- 8459 Views
- 3 replies
- 3 kudos
- 3 kudos
I think I sould have something missconfigured, the way I solved the problem was to re-create the workspace and start from scratch, it was a small one for testing proposes.
- 3 kudos
- 9047 Views
- 2 replies
- 2 kudos
AnalysisException: [UC_COMMAND_NOT_SUPPORTED] Spark higher-order functions are not supported in Unity Catalog.;
Hello,forecast_date = '2017-12-01' spark.conf.set('spark.sql.shuffle.partitions', 500 ) # generate forecast for this data forecasts = ( history .where(history.date < forecast_date) # limit training data to prior to our forecast date .groupBy...
- 9047 Views
- 2 replies
- 2 kudos
- 2 kudos
@Mr_K ApplyInPandas is a higher order function in Python. As of now, we do not support higher order functions in Unity Catalog. We do support direct calls made to python UDFs. Here is an example of how to reference UDFs in UC - https://docs.databrick...
- 2 kudos
- 2871 Views
- 1 replies
- 1 kudos
Does Unity Catalog on Azure require premium blob storage tier?
from the docs I read : Create a storage container where the metastore’s managed table data will be stored.This storage container must be in a Premium performance Azure Data Lake Storage Gen2 account in the same region as the workspaces you want to us...
- 2871 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @Carlos Restrepo​ We haven't heard from you since the last response from @Kaniz Fatma​ ​, and I was checking back to see if her suggestions helped you.Or else, If you have any solution, please share it with the community, as it can be helpful to o...
- 1 kudos