cancel
Showing results for 
Search instead for 
Did you mean: 
Data Governance
Join discussions on data governance practices, compliance, and security within the Databricks Community. Exchange strategies and insights to ensure data integrity and regulatory compliance.
cancel
Showing results for 
Search instead for 
Did you mean: 
Data + AI Summit 2024 - Data Governance

Forum Posts

joe_zhou
by New Contributor III
  • 5098 Views
  • 11 replies
  • 0 kudos

Resolved! Error: "Access validation failed" and error detail is "PERMISSION_DENIED: Cannot find shard endpoint for shard az-eastasia"

HiGetting the error message as below: when following the tutorial:Get started using Unity Catalog - Azure Databricks | Microsoft Learnin the section of "Create your first metastore and attach a workspace" with both storage and connector information f...

Databricks-create-metastore
  • 5098 Views
  • 11 replies
  • 0 kudos
Latest Reply
Andrew_Li
New Contributor III
  • 0 kudos

This is a bug, thanks for reporting it! Databricks engineers would address this soon.

  • 0 kudos
10 More Replies
Gustavo_Az
by Contributor
  • 4787 Views
  • 4 replies
  • 0 kudos

Resolved! Cant make Unity Catalog work on Azure

HelloI have created the metastore, linked it to the worskpace, my account is admin in Azure and Databricks, followed these instructions:link1link2link3But when I try to create a table (granting privileges to use catalog, create schemas and create tab...

error unity catalog error overlaps with managed storage
  • 4787 Views
  • 4 replies
  • 0 kudos
Latest Reply
Gustavo_Az
Contributor
  • 0 kudos

I finally made it work.It seems that although after configuring the metastore to work with a manged identity, the metastore must be upgraded with this guide.

  • 0 kudos
3 More Replies
Arpi
by New Contributor II
  • 1407 Views
  • 2 replies
  • 0 kudos

Delta table error

Hi,we are creating an external table in the Azure databricks but facing this error:org.apache.spark.sql.AnalysisException: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.meta...

  • 1407 Views
  • 2 replies
  • 0 kudos
Latest Reply
karthik_p
Esteemed Contributor
  • 0 kudos

@Priyanka Biswas​ is there a place where we can have updates on metastore upgrades or control plane individual components. usually we can only see control plane region wide status

  • 0 kudos
1 More Replies
Kayl669
by New Contributor III
  • 1343 Views
  • 2 replies
  • 1 kudos

Reassurance sought about behaviour of Databricks account SCIM connector

In my org we've got workspaces with a mixture of SCIM-provisioned and non-SCIM groups. These are all 'workspace local' groups. My identity provider is AAD.I've created a new workspace and want users in this workspace to be provided access only via ac...

  • 1343 Views
  • 2 replies
  • 1 kudos
Latest Reply
Debayan
Esteemed Contributor III
  • 1 kudos

Hi, Please refer to https://learn.microsoft.com/en-us/azure/databricks/administration-guide/users-groups/users and https://learn.microsoft.com/en-us/azure/databricks/administration-guide/users-groups/groups. Also, please note if you already have SCIM...

  • 1 kudos
1 More Replies
RicksDB
by Contributor II
  • 2990 Views
  • 3 replies
  • 2 kudos

Unity Catalog shared cluster & third-party1 libraries

Hello,I started testing Unity catalog and was surprised by the following mention in the documentation about shared clusters not supporting third-party1 libraries. Nevertheless, I tried doing imports and using pip install and both were working correct...

DB" data-fileid="0698Y00000Fb1noQAB
  • 2990 Views
  • 3 replies
  • 2 kudos
Latest Reply
MattG1
New Contributor II
  • 2 kudos

Hi Aviral-Bhardwaj-ji, I have seen a couple of mentions about Whitelisting of libraries on the Account but have struggled to find any detail about how to actually do this. Like RickDB (OP) I have been exploring the idea of 'whitelisting' libraries on...

  • 2 kudos
2 More Replies
Vgupta1
by New Contributor
  • 1422 Views
  • 3 replies
  • 0 kudos

Where mount info get stored in external hive metastore?

I have created a cluster which has advance options set for external hive metastore. I created a mount using sp for my data lake and then create a table using the mount . As per my knowledge the metadata of the table will be stored in sql db but i wou...

  • 1422 Views
  • 3 replies
  • 0 kudos
Latest Reply
User16725394280
Contributor II
  • 0 kudos

Hi @Vishal Gupta​ when we create a mount we store the mount info in a file in the DBFS root bucket.

  • 0 kudos
2 More Replies
parulpaul
by New Contributor III
  • 3667 Views
  • 6 replies
  • 9 kudos

I do not find "Create Catalog "and "Create Metastore" on Azure Databricks.

I am trying to set up Unity Catalog in Azure Databricks following this documentation Unity Catalog, but I do not find "Create Metastore" and "Create Metastore" on Databricks.Please help with the same, how can I enable Unity Catalog.

  • 3667 Views
  • 6 replies
  • 9 kudos
Latest Reply
parulpaul
New Contributor III
  • 9 kudos

This issue is resolved. Thank you @Camilla Lok​ and @Harun for the suggestions. It works fine after enabling Global administrators in Azure Active Directory .

  • 9 kudos
5 More Replies
rammy
by Contributor III
  • 2945 Views
  • 4 replies
  • 7 kudos

There is no response when invoking the REST service in Databricks notebook

I am trying to invoke a REST service from Databrcisks notebook using python programming. It is working fine before enabling Unity Catalog but after enabling it (In another environment) it is not working and never giving any response and not showing a...

  • 2945 Views
  • 4 replies
  • 7 kudos
Latest Reply
Prabakar
Esteemed Contributor III
  • 7 kudos

I suspect the problem could be caused by the cluster mode. What type of cluster or what access mode do you use for your cluster?

  • 7 kudos
3 More Replies
karthik_p
by Esteemed Contributor
  • 799 Views
  • 3 replies
  • 4 kudos

unable to open external table created under hive_metastore (Data view) in azure .

we have enabled unity catalog in Azure, we have a requirement to create external table in hive_metastore. we have configured ADLS Gen2 and by using different access methods that ADLS Gen 2 supports we have created external table. we are able to view ...

  • 799 Views
  • 3 replies
  • 4 kudos
Latest Reply
Anonymous
Not applicable
  • 4 kudos

@karthik p​ Unity Catalog is supported in DBR 11.2 and higher versions. If you attempt to create hive external tables on these DBR versions by default storage credential will be used for authentication to the storage account. You can create storage c...

  • 4 kudos
2 More Replies
BananaHotSauce
by New Contributor III
  • 1669 Views
  • 3 replies
  • 2 kudos

OracleDB shared Cluster / DBFS on Unity Catalog Cluster

Hello,im trying to use oracledb on my cluster and having issues using it.An init script on the cluster fixed this issue.Now, im trying to create a shared cluster with unity catalog and have oracledb in it.I encountered issues having this.- Init scrip...

  • 1669 Views
  • 3 replies
  • 2 kudos
Latest Reply
Anonymous
Not applicable
  • 2 kudos

@Chris Joshua Manuel​ The support for init script, library installation, Machine Learning, UDF on shared cluster is in preview. Please submit this form to onboard your workspace for improved shared clusters preview. https://docs.google.com/forms/d/e...

  • 2 kudos
2 More Replies
parthsalvi
by Contributor
  • 7793 Views
  • 3 replies
  • 6 kudos

UDF/UDAF/SQL Function not working in Unity catalog Error in DBR 10.4 LTS Shared mode

We're trying to use pandas_udf in DBR 10.4 LTS Shared mode but keep running in the below error :AnalysisException: UDF/UDAF/SQL functions is not supported in Unity Catalog;But in Single User mode above code works correctly.

image
  • 7793 Views
  • 3 replies
  • 6 kudos
Latest Reply
Hubert-Dudek
Esteemed Contributor III
  • 6 kudos

For me, it is like a bug as, in my opinion, that code should be independent of the unity catalog as it sparks code, and you don't register anything in the unity catalog. So please pass it to your databricks representative or as a support ticket.There...

  • 6 kudos
2 More Replies
evega
by New Contributor III
  • 6540 Views
  • 11 replies
  • 1 kudos

Azure Create Metastore button not available

Dear community, I want to configure the unity catalog with databricks, but the Create MetaStore button does not appear in my Azure Databricks workspace.In the community, I have seen that several users present the same situation ... What is the reason...

  • 6540 Views
  • 11 replies
  • 1 kudos
Latest Reply
evega
New Contributor III
  • 1 kudos

Hello Databricks Community, good news! I already found the solution: To login as an administrator, it is necessary to have the role of global administrator on the azure subscription, otherwise it will never appear.

  • 1 kudos
10 More Replies
oteng
by New Contributor III
  • 3064 Views
  • 2 replies
  • 0 kudos

Resolved! Schema browser for hive_metastore not working

Hi,I am not able to access the schemas inside the hive_metastore in the schema browser inside the SQL Editor in Databricks SQL. I tried on 2 databricks instances. One has Unity Catalog enabled and I am able to access the other catalogs/schemas but no...

  • 3064 Views
  • 2 replies
  • 0 kudos
Latest Reply
Aviral-Bhardwaj
Esteemed Contributor III
  • 0 kudos

this is something weird, can we connect and solve your issue

  • 0 kudos
1 More Replies
HemanthRatakond
by New Contributor II
  • 1172 Views
  • 2 replies
  • 2 kudos

Reading Athen table created on top of s3 in databricks

HI,we have databricks that use aws glue catalog as metastore, I am trying to read athena table which is created on top s3, I am getting following errorcom.databricks.backend.common.rpc.SparkDriverExceptions$SQLExecutionException: java.lang.RuntimeExc...

  • 1172 Views
  • 2 replies
  • 2 kudos
Latest Reply
HemanthRatakond
New Contributor II
  • 2 kudos

@Daniel Sahal​ at org.apache.hadoop.hive.ql.plan.TableDesc.getDeserializerClass(TableDesc.java:79) at org.apache.spark.sql.hive.execution.HiveTableScanExec.addColumnMetadataToConf(HiveTableScanExec.scala:127) at org.apache.spark.sql.hive.execution.Hi...

  • 2 kudos
1 More Replies
divya_k
by New Contributor II
  • 1191 Views
  • 3 replies
  • 4 kudos

Storage location of table access control while connecting to external hive metastore in unity catalog

I am a databricks administrator in azure databricI want to find out which storage location is used by data bricks to store the table access control while connecting to external hive metastore in unity catalog, is there any option to change that stora...

  • 1191 Views
  • 3 replies
  • 4 kudos
Latest Reply
Anonymous
Not applicable
  • 4 kudos

Hi @Ankitha V​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks!

  • 4 kudos
2 More Replies
Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!

Labels