cancel
Showing results for 
Search instead for 
Did you mean: 
Data Governance
Join discussions on data governance practices, compliance, and security within the Databricks Community. Exchange strategies and insights to ensure data integrity and regulatory compliance.
cancel
Showing results for 
Search instead for 
Did you mean: 
Data + AI Summit 2024 - Data Governance

Forum Posts

MadelynM
by Databricks Employee
  • 676 Views
  • 0 replies
  • 0 kudos

[Recap] Data + AI Summit 2024 - Data Governance | Navigate the explosion of AI, data and tools

Here's your Data + AI Summit 2024 - Data Governance recap as you navigate the explosion of AI, data and tools in efforts to build a flexible and scalable governance framework that spans your entire data and AI estate. Keynote: Evolving Data Governan...

Screenshot 2024-07-03 at 9.27.29 AM.png
  • 676 Views
  • 0 replies
  • 0 kudos
joe_zhou
by New Contributor III
  • 7082 Views
  • 11 replies
  • 0 kudos

Resolved! Error: "Access validation failed" and error detail is "PERMISSION_DENIED: Cannot find shard endpoint for shard az-eastasia"

HiGetting the error message as below: when following the tutorial:Get started using Unity Catalog - Azure Databricks | Microsoft Learnin the section of "Create your first metastore and attach a workspace" with both storage and connector information f...

Databricks-create-metastore
  • 7082 Views
  • 11 replies
  • 0 kudos
Latest Reply
Andrew_Li
New Contributor III
  • 0 kudos

This is a bug, thanks for reporting it! Databricks engineers would address this soon.

  • 0 kudos
10 More Replies
Gustavo_Az
by Contributor
  • 6457 Views
  • 4 replies
  • 0 kudos

Resolved! Cant make Unity Catalog work on Azure

HelloI have created the metastore, linked it to the worskpace, my account is admin in Azure and Databricks, followed these instructions:link1link2link3But when I try to create a table (granting privileges to use catalog, create schemas and create tab...

error unity catalog error overlaps with managed storage
  • 6457 Views
  • 4 replies
  • 0 kudos
Latest Reply
Gustavo_Az
Contributor
  • 0 kudos

I finally made it work.It seems that although after configuring the metastore to work with a manged identity, the metastore must be upgraded with this guide.

  • 0 kudos
3 More Replies
Arpi
by New Contributor II
  • 1947 Views
  • 2 replies
  • 0 kudos

Delta table error

Hi,we are creating an external table in the Azure databricks but facing this error:org.apache.spark.sql.AnalysisException: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.meta...

  • 1947 Views
  • 2 replies
  • 0 kudos
Latest Reply
karthik_p
Esteemed Contributor
  • 0 kudos

@Priyanka Biswas​ is there a place where we can have updates on metastore upgrades or control plane individual components. usually we can only see control plane region wide status

  • 0 kudos
1 More Replies
Kayl669
by New Contributor III
  • 1789 Views
  • 2 replies
  • 1 kudos

Reassurance sought about behaviour of Databricks account SCIM connector

In my org we've got workspaces with a mixture of SCIM-provisioned and non-SCIM groups. These are all 'workspace local' groups. My identity provider is AAD.I've created a new workspace and want users in this workspace to be provided access only via ac...

  • 1789 Views
  • 2 replies
  • 1 kudos
Latest Reply
Debayan
Databricks Employee
  • 1 kudos

Hi, Please refer to https://learn.microsoft.com/en-us/azure/databricks/administration-guide/users-groups/users and https://learn.microsoft.com/en-us/azure/databricks/administration-guide/users-groups/groups. Also, please note if you already have SCIM...

  • 1 kudos
1 More Replies
RicksDB
by Contributor II
  • 3642 Views
  • 3 replies
  • 2 kudos

Unity Catalog shared cluster & third-party1 libraries

Hello,I started testing Unity catalog and was surprised by the following mention in the documentation about shared clusters not supporting third-party1 libraries. Nevertheless, I tried doing imports and using pip install and both were working correct...

DB" data-fileid="0698Y00000Fb1noQAB
  • 3642 Views
  • 3 replies
  • 2 kudos
Latest Reply
MattG1
New Contributor II
  • 2 kudos

Hi Aviral-Bhardwaj-ji, I have seen a couple of mentions about Whitelisting of libraries on the Account but have struggled to find any detail about how to actually do this. Like RickDB (OP) I have been exploring the idea of 'whitelisting' libraries on...

  • 2 kudos
2 More Replies
Vgupta1
by New Contributor
  • 1859 Views
  • 3 replies
  • 0 kudos

Where mount info get stored in external hive metastore?

I have created a cluster which has advance options set for external hive metastore. I created a mount using sp for my data lake and then create a table using the mount . As per my knowledge the metadata of the table will be stored in sql db but i wou...

  • 1859 Views
  • 3 replies
  • 0 kudos
Latest Reply
User16725394280
Contributor II
  • 0 kudos

Hi @Vishal Gupta​ when we create a mount we store the mount info in a file in the DBFS root bucket.

  • 0 kudos
2 More Replies
parulpaul
by New Contributor III
  • 4838 Views
  • 6 replies
  • 9 kudos

I do not find "Create Catalog "and "Create Metastore" on Azure Databricks.

I am trying to set up Unity Catalog in Azure Databricks following this documentation Unity Catalog, but I do not find "Create Metastore" and "Create Metastore" on Databricks.Please help with the same, how can I enable Unity Catalog.

  • 4838 Views
  • 6 replies
  • 9 kudos
Latest Reply
parulpaul
New Contributor III
  • 9 kudos

This issue is resolved. Thank you @Camilla Lok​ and @Harun for the suggestions. It works fine after enabling Global administrators in Azure Active Directory .

  • 9 kudos
5 More Replies
rammy
by Contributor III
  • 3838 Views
  • 4 replies
  • 7 kudos

There is no response when invoking the REST service in Databricks notebook

I am trying to invoke a REST service from Databrcisks notebook using python programming. It is working fine before enabling Unity Catalog but after enabling it (In another environment) it is not working and never giving any response and not showing a...

  • 3838 Views
  • 4 replies
  • 7 kudos
Latest Reply
Prabakar
Databricks Employee
  • 7 kudos

I suspect the problem could be caused by the cluster mode. What type of cluster or what access mode do you use for your cluster?

  • 7 kudos
3 More Replies
karthik_p
by Esteemed Contributor
  • 1160 Views
  • 3 replies
  • 4 kudos

unable to open external table created under hive_metastore (Data view) in azure .

we have enabled unity catalog in Azure, we have a requirement to create external table in hive_metastore. we have configured ADLS Gen2 and by using different access methods that ADLS Gen 2 supports we have created external table. we are able to view ...

  • 1160 Views
  • 3 replies
  • 4 kudos
Latest Reply
Anonymous
Not applicable
  • 4 kudos

@karthik p​ Unity Catalog is supported in DBR 11.2 and higher versions. If you attempt to create hive external tables on these DBR versions by default storage credential will be used for authentication to the storage account. You can create storage c...

  • 4 kudos
2 More Replies
BananaHotSauce
by New Contributor III
  • 2223 Views
  • 3 replies
  • 2 kudos

OracleDB shared Cluster / DBFS on Unity Catalog Cluster

Hello,im trying to use oracledb on my cluster and having issues using it.An init script on the cluster fixed this issue.Now, im trying to create a shared cluster with unity catalog and have oracledb in it.I encountered issues having this.- Init scrip...

  • 2223 Views
  • 3 replies
  • 2 kudos
Latest Reply
Anonymous
Not applicable
  • 2 kudos

@Chris Joshua Manuel​ The support for init script, library installation, Machine Learning, UDF on shared cluster is in preview. Please submit this form to onboard your workspace for improved shared clusters preview. https://docs.google.com/forms/d/e...

  • 2 kudos
2 More Replies
parthsalvi
by Contributor
  • 9153 Views
  • 3 replies
  • 6 kudos

UDF/UDAF/SQL Function not working in Unity catalog Error in DBR 10.4 LTS Shared mode

We're trying to use pandas_udf in DBR 10.4 LTS Shared mode but keep running in the below error :AnalysisException: UDF/UDAF/SQL functions is not supported in Unity Catalog;But in Single User mode above code works correctly.

image
  • 9153 Views
  • 3 replies
  • 6 kudos
Latest Reply
Hubert-Dudek
Esteemed Contributor III
  • 6 kudos

For me, it is like a bug as, in my opinion, that code should be independent of the unity catalog as it sparks code, and you don't register anything in the unity catalog. So please pass it to your databricks representative or as a support ticket.There...

  • 6 kudos
2 More Replies
evega
by New Contributor III
  • 9827 Views
  • 11 replies
  • 1 kudos

Azure Create Metastore button not available

Dear community, I want to configure the unity catalog with databricks, but the Create MetaStore button does not appear in my Azure Databricks workspace.In the community, I have seen that several users present the same situation ... What is the reason...

  • 9827 Views
  • 11 replies
  • 1 kudos
Latest Reply
evega
New Contributor III
  • 1 kudos

Hello Databricks Community, good news! I already found the solution: To login as an administrator, it is necessary to have the role of global administrator on the azure subscription, otherwise it will never appear.

  • 1 kudos
10 More Replies
oteng
by New Contributor III
  • 4028 Views
  • 2 replies
  • 0 kudos

Resolved! Schema browser for hive_metastore not working

Hi,I am not able to access the schemas inside the hive_metastore in the schema browser inside the SQL Editor in Databricks SQL. I tried on 2 databricks instances. One has Unity Catalog enabled and I am able to access the other catalogs/schemas but no...

  • 4028 Views
  • 2 replies
  • 0 kudos
Latest Reply
Aviral-Bhardwaj
Esteemed Contributor III
  • 0 kudos

this is something weird, can we connect and solve your issue

  • 0 kudos
1 More Replies
HemanthRatakond
by New Contributor II
  • 1770 Views
  • 2 replies
  • 2 kudos

Reading Athen table created on top of s3 in databricks

HI,we have databricks that use aws glue catalog as metastore, I am trying to read athena table which is created on top s3, I am getting following errorcom.databricks.backend.common.rpc.SparkDriverExceptions$SQLExecutionException: java.lang.RuntimeExc...

  • 1770 Views
  • 2 replies
  • 2 kudos
Latest Reply
HemanthRatakond
New Contributor II
  • 2 kudos

@Daniel Sahal​ at org.apache.hadoop.hive.ql.plan.TableDesc.getDeserializerClass(TableDesc.java:79) at org.apache.spark.sql.hive.execution.HiveTableScanExec.addColumnMetadataToConf(HiveTableScanExec.scala:127) at org.apache.spark.sql.hive.execution.Hi...

  • 2 kudos
1 More Replies
divya_k
by New Contributor II
  • 1558 Views
  • 2 replies
  • 4 kudos

Storage location of table access control while connecting to external hive metastore in unity catalog

I am a databricks administrator in azure databricI want to find out which storage location is used by data bricks to store the table access control while connecting to external hive metastore in unity catalog, is there any option to change that stora...

  • 1558 Views
  • 2 replies
  • 4 kudos
Latest Reply
Anonymous
Not applicable
  • 4 kudos

Hi @Ankitha V​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks!

  • 4 kudos
1 More Replies

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels