- 252 Views
- 3 replies
- 3 kudos
child Subqueries/sub-statements history metrics, from a parent [CALL...] statement in QueryHistory
Hi,I cannot find so far a way to get programmatically (SQL/Python) the Subqueries(/Sub-statements) executions history records, shown in ADBricks UI Query History/Profile, that were executed during a TaskRun of Job, as shown in [red boxes] on the atta...
- 252 Views
- 3 replies
- 3 kudos
- 3 kudos
Hi @ADBricksExplore, The sub-statements panel you see in the Query History UI (the child queries spawned by a CALL or multi-statement execution) is rendered from an internal GraphQL endpoint that is not part of the public Databricks API surface. That...
- 3 kudos
- 310 Views
- 2 replies
- 1 kudos
Hive metastore CANNOT access storage
I'm new to Azure Databricks and I'm facing an issue when trying to create a schema or table that points to my Azure Storage account. I keep getting this error:```[EXTERNAL_METASTORE_CLIENT_ERROR.OPERATION_FAILED] Client operation failed: org.apache.h...
- 310 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @cobba16, This is a common scenario on Azure Databricks when using the legacy hive_metastore with external storage. The root cause is that dbutils.fs.ls and Spark read operations use the credentials you set in the Spark session config, but the Hiv...
- 1 kudos
- 265 Views
- 1 replies
- 0 kudos
Grant permissions to existing catalogs/schemas using Databricks Asset Bundles
Hi,I’m trying to use Databricks Asset Bundles (DAB) to assign Unity Catalog grants to catalogs and schemas that already exist in my workspace.These catalogs and schemas were not originally created through DAB, but I would now like to manage their gra...
- 265 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @vjussiiih, Let me walk you through this. You are correct that DABs treat schemas and catalogs defined under "resources" as fully managed resources, which means they attempt to create them on first deploy and manage their full lifecycle. This is w...
- 0 kudos
- 102 Views
- 1 replies
- 1 kudos
Unity Catalog Data Classification Dashboard: When Does the “User Access” Column Get Populated?
Hi everyone,I’m currently testing the Data Classification feature in Databricks Unity Catalog.On the dashboard after classification completes, there’s a column called **“User Access.”** Based on the description, it seems to show the number of users w...
- 102 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @r_w_, Appreciate you sharing the details. the "User Access" column in the data classification results view is one of those features that is not thoroughly covered in the public documentation yet, so I understand the confusion. Here is what I have...
- 1 kudos
- 89 Views
- 1 replies
- 0 kudos
Turn off data export functionality
Is there a current way to limit data exporting. Meaning not allowing users export data from a db objects to csv, excel, or copying results to clipboard. I see that currently exporting/downloading data can be done at workspace level, but I am proposin...
- 89 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @bigdatabase, As of today, Databricks lets you control downloading notebook results only at the workspace level. As you pointed out, a workspace admin can disable downloading results from the notebook for all users in that workspace. However, ther...
- 0 kudos
- 11032 Views
- 6 replies
- 0 kudos
Unity Catalog- Informatica Cloud Data Governance & Catalog Integration
We are attempting to create a comprehensive Data Governance solution using Unity Catalog & INFA CDGC tool. The objective is to onboard the business assets on the Informatica DG platform and use the unity catalog to trace technical assets and lineage ...
- 11032 Views
- 6 replies
- 0 kudos
- 0 kudos
@TanushM -- We are also planning to do the same. However we are struggling with the volume of tables inside some schemas in a catalog. Can you suggest some of the best practices you have come across while integrating Unity catalog to CDGC?
- 0 kudos
- 2945 Views
- 5 replies
- 4 kudos
Resolved! Will Unity catalog come to Europe?
It is our understanding that enabling unity catalog means that some metadata (for example usernames) will be sent to the USA for processing/storage. This is unfortunately a deal-breaker for us, and we need the data to reside solely in Europe for comp...
- 2945 Views
- 5 replies
- 4 kudos
- 4 kudos
Usernames have always been stored centrally in the US. That's similar to most global services for the IAM Layer. The only thing additional that gets stored in the US as part of UC is the metastore names. you can check this link ( https://azure.micros...
- 4 kudos
- 146 Views
- 1 replies
- 0 kudos
Resolved! Unity catalog management
How is Unity Catalog managed in real time at an enterprise scale, including workspace-level restrictions, privilege-based ACLs, row- and column-level security, ABAC, and tag-driven governance, and which languages or tools are used to manage the entir...
- 146 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @APJESK ,The most common approach I've seen in enterprise is to use terraform to govern Unity Catalog. Below you can find a good series of articles that introduce this concept:https://pl.seequality.net/terra-dbx-p1/Databricks terraform provider is...
- 0 kudos
- 487 Views
- 2 replies
- 2 kudos
Resolved! Managing Spark Declarative Pipelines Permissions
We are currently managing our permissions via Terraform (including cluster creation, UC governance, etc.). We have a specific `data_engineer` role, and we need everyone with this role to be able to view and manage all of our SDPs.The Issue: Currently...
- 487 Views
- 2 replies
- 2 kudos
- 2 kudos
Our Solution: We moved job and pipeline permissions to DAB configuration files for streamlined enforcement. Terraform will remain the source of truth for workspace-level permissions only.
- 2 kudos
- 545 Views
- 3 replies
- 6 kudos
Resolved! Issues while running SYNC SCHEMA (HIVE-6384)
Hi All, We are in a process of migrating to UC from HMS and also changing the DBR from 11.3 to 16.4 LTS. When i am running sync schema from HMS to UC i am getting "Parquet does not support date. See HIVE-6384" error. Trying to resolve this, i have di...
- 545 Views
- 3 replies
- 6 kudos
- 6 kudos
@yashojhaIf you are doing HMS to UC migration. It is worth checking UCX an automated Unity catalog migration tool.https://github.com/databrickslabs/ucx
- 6 kudos
- 794 Views
- 3 replies
- 2 kudos
Resolved! Databricks Apps - Automating Unity Catalog Privileges for Databricks Apps Service Principals
I am using Databricks Apps, and I want to grant privileges to the auto-generated service principal from the app to read specific Unity Catalog schemas. I can't find a way to do this automatically using Asset Bundles.I prefer to query using a service ...
- 794 Views
- 3 replies
- 2 kudos
- 2 kudos
Why not to run a databricks cli script for first assign proper permissions (https://docs.databricks.com/aws/en/dev-tools/cli/reference/account-commands) and then run DAB? This was my approach to deploy some components not already available in previou...
- 2 kudos
- 380 Views
- 1 replies
- 1 kudos
Resolved! Mouse movement is not available on Databricks Dashboards in case of many pages
Hello Team, We are building different KPIs on the Databricks Dashboard, the issue is: We have many pages which are part of the Dashboard, they are working but the issue is we are not able to navigate the pages with mouse if they are moving out of the...
- 380 Views
- 1 replies
- 1 kudos
- 1 kudos
Hello @saravjeet! You can check out this similar post: Databricks Dashboard Issue: No Mouse-Based Navigation When Dashboard Tabs Exceed the Top Ribbon If you’re using a mouse, you can navigate horizontally by: Holding Shift and scrolling with the mou...
- 1 kudos
- 468 Views
- 3 replies
- 2 kudos
Data Profile on a table is not securable object
Data Profile on a table is not a securable object in Unity Catalog or at Workspace level. This make the management of Data Profiles difficult for workspace admins.Why isn’t “profile” a securable object in Databricks? It makes sense to require “Manage...
- 468 Views
- 3 replies
- 2 kudos
- 2 kudos
Data profiling is a configuration you attach to a table. The output metric tables are Unity Catalog objects that you choose the schema for and can govern with UC privileges like any other table. The profile’s configuration and refresh history are wor...
- 2 kudos
- 68293 Views
- 20 replies
- 11 kudos
Unity Catalog Shared Access Mode - dbutils.notebook.entry_point...getContext() not whitelisted
we are switching over to Unity Catalog and attempting to confirm the ability to run our existing notebooks. I have created a new Shared Unity Catalog Cluster and ran the notebook using the new cluster. Ran into an error attempting to execute a prin...
- 68293 Views
- 20 replies
- 11 kudos
- 11 kudos
To replace dbutils.notebook.entry_point.getDbutils().notebook().getContext().toJson() and related APIs try:from dbruntime.databricks_repl_context import get_context token = get_context().apiToken host = get_context().browserHostName get_context().__d...
- 11 kudos
- 987 Views
- 1 replies
- 2 kudos
Resolved! Unity Catalog functions
Hi community, Recent days I created a SQL function into my unity catalog for my trusted assets in my genie space, I gave permission to my users accounts for they saw these function, however yesterday I modified my functions using the command "CREATE ...
- 987 Views
- 1 replies
- 2 kudos
- 2 kudos
Greetings @Dulce42 , this is a known gotcha with Unity Catalog functions: updating a function with CREATE OR REPLACE FUNCTION currently replaces the object and drops its grants, so downstream users lose EXECUTE permission and need to be re-granted. ...
- 2 kudos