we are switching over to Unity Catalog and attempting to confirm the ability to run our existing notebooks. I have created a new Shared Unity Catalog Cluster and ran the notebook using the new cluster. Ran into an error attempting to execute a prin...
I faced the same issue when switching to Shared Access cluster and found that there is a possibility to run :dbutils.notebook.entry_point.getDbutils().notebook().getContext().safeToJson()Hope this helps
Databricks autocomplete is checking the hive_metastore catalog when I enter the schema_name and table_name in the notebook although we have other default catalog set at the workspace level. How to make Databricks autocomplete use that catalog when en...
Hi @Kaniz ! Thanks for your answer. I forgot to mention that we already have this set up at the cluster level (using spark.databricks.sql.initial.catalog.name variable) besides setting this on the workspace level in the workspace settings but none of...
Hi,We are rolling out Unity in our organization and while playing around I had set some tags on a test catalog.After removing this catalog, the tag keys I used are still present in our Unity environment and I am unable to find where they are stored t...
I'm trying to use bamboolib on the 11.0 recommended DBR. I am using the option "Databricks: Load database table" The databases and tables that are on the unity catalog do not populate the dropdown boxes. How can I fix this?
Hello @k.b. Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks!
We are looking to send messages using databricks to an azure service bus topic and would like to connect to the service bus using a managed identity vs a connection string. Is this possible in databricks. The only thing I could find regarding datab...
To be honest, the Databricks team sucks. They should add a simple Identity tab under the Databricks workspace resource as it is done for other Microsoft services. They don't think about making product easier to maintain and configure, they look from ...
Is it possible to provide fine grained control (folder level/file level) for a given volume?I have two SCIM integrated groups who have read volume access at the catalog level, but those two groups need different permissions on a lower level. Preferab...
Can you define the external location at the Landing level and create two Volumes one for PDF and other for CSV and provide access to the respective groups 1 and 2.
I have been using "rdd.flatMap(lambda x:x)" for a while to create lists from columns however after I have changed the cluster to a Shared acess mode (to use unity catalog) I get the following error: py4j.security.Py4JSecurityException: Method public ...
Faced this issue multiple times.Solution:1. Don't use Shared Cluster or cluster without Unity Catalog enabled for running 'rdd' queries on Databricks.2. Instead create a Personal Cluster (Single User) with basic configuration and with Unity Catalog e...
I saw a really nice article (https://www.databricks.com/blog/announcing-public-preview-volumes-databricks-unity-catalog) on the incorporation of various elements of data lineage within Unity Catalog. In my own exploration, I've been able to replicate...
Hi @23940829381, Thank you for sharing your interest in data lineage within Unity Catalog! It’s a powerful feature that allows you to track the flow of data across various elements in your Databricks environment.
Let’s delve into this further:
Un...
We're thrilled to introduce UCX v0.15.0, packed with cutting-edge features, enhancements, and deprecations that'll take your upgrading experience to new heights!
Release Highlights:
AWS S3 support: We've added AWS S3 support to the migrate-location...
We are attempting to create a comprehensive Data Governance solution using Unity Catalog & INFA CDGC tool. The objective is to onboard the business assets on the Informatica DG platform and use the unity catalog to trace technical assets and lineage ...
Yes, Informatica now has native integration to Unity Catalog with its cloud data catalogue and governance service. You can scan Unity catalog Metadata and propagate this in CDGC and apply policies, business terms, overlay data quality scores, display...
Hi All, Can anyone point me to either documentation or personally tried and tested method of backing up (and restoring) Unity Catalog and its associated managed tables? We're running on Azure and using ADLS Gen2.Regards,Ashley
Our UC managed tables are stored on prod ADLS storage which is different from UC root storage account. So what's the best way to backup and restore UC managed tables into different region? One option is deep clone tables, copy ADLS folders to an...
I faced with error "[RequestId=b567c3bb-05fc-4ac8-b4db-2d826b2c00be ErrorClass=QUOTA_EXCEEDED] Cannot create 1 Table(s) in Metastore 95157916-5539-4dc4-a7d6-e9a854ba0591 (estimated count: 100088, limit: 100000). " but I didn't see here this limit ...
Hi, Databricks CommunityI am currently plan to do migrate of existing metastore (not in a desired account and name) to a new one (a different desired account) within the same region. I understand it is not as straightforward it might be and complicat...
Hi Databricks community! Exciting news: we open-sourced PACE (Policy As Code Engine) and launched on Product Hunt, and we'd love your input!PACE offers an abstraction on Unity Catalog data access policies, translating YAML/JSON policy definitions int...
I’m currently trying to create a Foreign Catalog based on a Connection object of type SQLSERVER. This would allow me to directly access our on-premises MS SQL database from within Azure Databricks using Unity Catalog.As I’m part of a large organizati...
Hi @MiroFuoli, Firstly, ensuring you’re using the correct versions of dbt-core and dbt-databricks is important. There have been instances where specific versions of these packages have caused issues. You might want to try using an older version, such...
