Here's your Data + AI Summit 2024 - Data Governance recap as you navigate the explosion of AI, data and tools in efforts to build a flexible and scalable governance framework that spans your entire data and AI estate. Keynote: Evolving Data Governan...
Hello,I started testing Unity catalog and was surprised by the following mention in the documentation about shared clusters not supporting third-party1 libraries. Nevertheless, I tried doing imports and using pip install and both were working correct...
Hi Aviral-Bhardwaj-ji, I have seen a couple of mentions about Whitelisting of libraries on the Account but have struggled to find any detail about how to actually do this. Like RickDB (OP) I have been exploring the idea of 'whitelisting' libraries on...
I have created a cluster which has advance options set for external hive metastore. I created a mount using sp for my data lake and then create a table using the mount . As per my knowledge the metadata of the table will be stored in sql db but i wou...
Hi @Vishal Gupta​ when we create a mount we store the mount info in a file in the DBFS root bucket.
I am trying to invoke a REST service from Databrcisks notebook using python programming. It is working fine before enabling Unity Catalog but after enabling it (In another environment) it is not working and never giving any response and not showing a...
I suspect the problem could be caused by the cluster mode. What type of cluster or what access mode do you use for your cluster?
we have enabled unity catalog in Azure, we have a requirement to create external table in hive_metastore. we have configured ADLS Gen2 and by using different access methods that ADLS Gen 2 supports we have created external table. we are able to view ...
@karthik p​ Unity Catalog is supported in DBR 11.2 and higher versions. If you attempt to create hive external tables on these DBR versions by default storage credential will be used for authentication to the storage account. You can create storage c...
Hello,im trying to use oracledb on my cluster and having issues using it.An init script on the cluster fixed this issue.Now, im trying to create a shared cluster with unity catalog and have oracledb in it.I encountered issues having this.- Init scrip...
@Chris Joshua Manuel​ The support for init script, library installation, Machine Learning, UDF on shared cluster is in preview. Please submit this form to onboard your workspace for improved shared clusters preview. https://docs.google.com/forms/d/e...
We're trying to use pandas_udf in DBR 10.4 LTS Shared mode but keep running in the below error :AnalysisException: UDF/UDAF/SQL functions is not supported in Unity Catalog;But in Single User mode above code works correctly.
For me, it is like a bug as, in my opinion, that code should be independent of the unity catalog as it sparks code, and you don't register anything in the unity catalog. So please pass it to your databricks representative or as a support ticket.There...
Hi,I am not able to access the schemas inside the hive_metastore in the schema browser inside the SQL Editor in Databricks SQL. I tried on 2 databricks instances. One has Unity Catalog enabled and I am able to access the other catalogs/schemas but no...
HI,we have databricks that use aws glue catalog as metastore, I am trying to read athena table which is created on top s3, I am getting following errorcom.databricks.backend.common.rpc.SparkDriverExceptions$SQLExecutionException: java.lang.RuntimeExc...
@Daniel Sahal​ at org.apache.hadoop.hive.ql.plan.TableDesc.getDeserializerClass(TableDesc.java:79) at org.apache.spark.sql.hive.execution.HiveTableScanExec.addColumnMetadataToConf(HiveTableScanExec.scala:127) at org.apache.spark.sql.hive.execution.Hi...
I am a databricks administrator in azure databricI want to find out which storage location is used by data bricks to store the table access control while connecting to external hive metastore in unity catalog, is there any option to change that stora...
Hi @Ankitha V​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks!
I have the administrator role, but I can't find where in the interface to enable access tokens. I created a single workspace. I haven't created a unity catalog or anything yet. Please advice. As well is there any best practice documentation on creati...
hey @Roberto Martin​ acess token will be generated by the admin only , as you have the admin access , then open your workspace see the right side there is the option of user settings click on that you will get another window , from where we can crea...
Hi,when i create external location on unity catalog. even if i dont provide any user grants, i have an write access to the s3 bucket from attached workspace.i created external location xxxxi didnt allow any permissions on it on a workspace that is co...
@Amit Cahanovich​ : Could you please share your config details,Which DBR version ?Cluster has any instance profile?
What are the best practices when using DLT with a unity catalog aware workspace since it is not supported at the moment?Do you create a copy of every table you need in the hive metastore in order to reference them? Or is it recommended to wait until ...
Hi @E H​ , the best way to manage this for the time being is to limit DLT to logic that is considered 'internal' i.e., leveraging the default hive metastore. For now at least, this should be treated as a transient landing zone. From here you can refi...
Without streaming everything works fine. When I try to set up some example structured streaming in my company's new Databricks workspace (we're trying it out) I can't seem to figure out the correct way to save it/write to storage. Please see the pict...
Hi, This can be an issue where Storage Blob Data contributor role is missing. Please refer: https://docs.databricks.com/external-data/azure-storage.html
Hi,I've got Unity Catalog working but i cant create an external Table.We have several Workspaces (with customer managed VPCs), all with Private Link connected to the Control Plane. Our Data S3 Buckets are secured via Bucket Policy (in addition to KM...
Hi, Could you please confirm if you have followed this: https://docs.databricks.com/data-governance/unity-catalog/get-started.html , also, please confirm the exact error received?
Hi there,I am facing an issue when setting up the metastore.I use an account admin user, and the user is part of a group called Unity admin.The Unity admin group contains 2 users: me and another user, also account admin.Without Unity catalog, for now...
Hi, Could you please refer https://docs.databricks.com/data-governance/unity-catalog/manage-privileges/index.html#transfer-ownership, please let us know if this helps.
