Hello,I started testing Unity catalog and was surprised by the following mention in the documentation about shared clusters not supporting third-party1 libraries. Nevertheless, I tried doing imports and using pip install and both were working correct...
Hi Aviral-Bhardwaj-ji, I have seen a couple of mentions about Whitelisting of libraries on the Account but have struggled to find any detail about how to actually do this. Like RickDB (OP) I have been exploring the idea of 'whitelisting' libraries on...
I have created a cluster which has advance options set for external hive metastore. I created a mount using sp for my data lake and then create a table using the mount . As per my knowledge the metadata of the table will be stored in sql db but i wou...
I am trying to set up Unity Catalog in Azure Databricks following this documentation Unity Catalog, but I do not find "Create Metastore" and "Create Metastore" on Databricks.Please help with the same, how can I enable Unity Catalog.
This issue is resolved. Thank you @Camilla Lok​ and @Harun for the suggestions. It works fine after enabling Global administrators in Azure Active Directory .
I am trying to invoke a REST service from Databrcisks notebook using python programming. It is working fine before enabling Unity Catalog but after enabling it (In another environment) it is not working and never giving any response and not showing a...
we have enabled unity catalog in Azure, we have a requirement to create external table in hive_metastore. we have configured ADLS Gen2 and by using different access methods that ADLS Gen 2 supports we have created external table. we are able to view ...
@karthik p​ Unity Catalog is supported in DBR 11.2 and higher versions. If you attempt to create hive external tables on these DBR versions by default storage credential will be used for authentication to the storage account. You can create storage c...
Hello,im trying to use oracledb on my cluster and having issues using it.An init script on the cluster fixed this issue.Now, im trying to create a shared cluster with unity catalog and have oracledb in it.I encountered issues having this.- Init scrip...
@Chris Joshua Manuel​ The support for init script, library installation, Machine Learning, UDF on shared cluster is in preview. Please submit this form to onboard your workspace for improved shared clusters preview. https://docs.google.com/forms/d/e...
We're trying to use pandas_udf in DBR 10.4 LTS Shared mode but keep running in the below error :AnalysisException: UDF/UDAF/SQL functions is not supported in Unity Catalog;But in Single User mode above code works correctly.
For me, it is like a bug as, in my opinion, that code should be independent of the unity catalog as it sparks code, and you don't register anything in the unity catalog. So please pass it to your databricks representative or as a support ticket.There...
Dear community, I want to configure the unity catalog with databricks, but the Create MetaStore button does not appear in my Azure Databricks workspace.In the community, I have seen that several users present the same situation ... What is the reason...
Hello Databricks Community, good news! I already found the solution: To login as an administrator, it is necessary to have the role of global administrator on the azure subscription, otherwise it will never appear.
Hi,I am not able to access the schemas inside the hive_metastore in the schema browser inside the SQL Editor in Databricks SQL. I tried on 2 databricks instances. One has Unity Catalog enabled and I am able to access the other catalogs/schemas but no...
HI,we have databricks that use aws glue catalog as metastore, I am trying to read athena table which is created on top s3, I am getting following errorcom.databricks.backend.common.rpc.SparkDriverExceptions$SQLExecutionException: java.lang.RuntimeExc...
@Daniel Sahal​ at org.apache.hadoop.hive.ql.plan.TableDesc.getDeserializerClass(TableDesc.java:79) at org.apache.spark.sql.hive.execution.HiveTableScanExec.addColumnMetadataToConf(HiveTableScanExec.scala:127) at org.apache.spark.sql.hive.execution.Hi...
I am a databricks administrator in azure databricI want to find out which storage location is used by data bricks to store the table access control while connecting to external hive metastore in unity catalog, is there any option to change that stora...
Hi @Ankitha V​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks!
I have the administrator role, but I can't find where in the interface to enable access tokens. I created a single workspace. I haven't created a unity catalog or anything yet. Please advice. As well is there any best practice documentation on creati...
hey @Roberto Martin​ acess token will be generated by the admin only , as you have the admin access , then open your workspace see the right side there is the option of user settings click on that you will get another window , from where we can crea...
Hi,when i create external location on unity catalog. even if i dont provide any user grants, i have an write access to the s3 bucket from attached workspace.i created external location xxxxi didnt allow any permissions on it on a workspace that is co...
What are the best practices when using DLT with a unity catalog aware workspace since it is not supported at the moment?Do you create a copy of every table you need in the hive metastore in order to reference them? Or is it recommended to wait until ...
Hi @E H​ , the best way to manage this for the time being is to limit DLT to logic that is considered 'internal' i.e., leveraging the default hive metastore. For now at least, this should be treated as a transient landing zone. From here you can refi...
Without streaming everything works fine. When I try to set up some example structured streaming in my company's new Databricks workspace (we're trying it out) I can't seem to figure out the correct way to save it/write to storage. Please see the pict...
Hi, This can be an issue where Storage Blob Data contributor role is missing. Please refer: https://docs.databricks.com/external-data/azure-storage.html
