Administration & Architecture
Explore discussions on Databricks administration, deployment strategies, and architectural best prac...
1389 Posts
Activity in Databricks Platform Discussions
- 1321 Views
- 10 replies
- 1 kudos
Unable to Enable Unity Catalog – Azure Managed Identity Credential Not Found
Hi Databricks Community,I am trying to enable Unity Catalog in my Azure Databricks account. As part of the setup, I created an Azure Databricks Access Connector and assigned its managed identity the required permissions on my ADLS Gen2 storage accoun...
- 1321 Views
- 10 replies
- 1 kudos
Latest Reply
- 1 kudos
I hit the same error.When creating a Unity Catalog metastore from the web UI of the Azure Databricks Account Console, the following error appeared during Access Connector validation, despite having all the Azure configuration correctly set up (manage...
- 1 kudos
- 70 Views
- 3 replies
- 0 kudos
Lakeflow connect CT pipeline keeps running multiple connections in source
I'm working on an ingestion pipeline. I was adding tables to it gradually so that we can monitor the load on our prod server side by side and with my last set of tables added (with them, all the heavy duty tables were inside that pipeline), the load ...
- 70 Views
- 3 replies
- 0 kudos
Latest Reply
- 0 kudos
For a standard connector, the gateway continuously extracts snapshots, change logs, and metadata, while a separately scheduled serverless pipeline applies staged data (connector components). Changing the serverless pipeline schedule therefore does no...
- 0 kudos
- 52 Views
- 1 replies
- 1 kudos
UC Delta API: managed-table READ_WRITE credentials rejected despite explicit User-Agent
Hi everyone,I’m developing HarborSQL, an open-source SQL engine using Apache DataFusion and delta-rs. I’m working on managed Delta table writes through the UC Delta API with catalog commits.Using the same workspace, managed table, and user identity:-...
- 52 Views
- 1 replies
- 1 kudos
Latest Reply
- 1 kudos
Hi @ablanchard .The behavior you're seeing is expected for the Delta protocol-level credential endpoint (/api/2.1/unity-catalog/delta/v1/.../credentials). A few things to note:Managed Delta table writes from external Delta clients are in Public Previ...
- 1 kudos
- 110 Views
- 3 replies
- 4 kudos
Feature enablement for Foundation Model Unity Catalog permissions
I am trying to restrict workspace users' access to Databricks Foundation Models using the guidance in the Foundation Model Unity Catalog Permissions documentation.I have revoked EXECUTE permission for all users from the system.ai schema. However, wor...
- 110 Views
- 3 replies
- 4 kudos
Latest Reply
- 4 kudos
It persists after enablement. The legacy databricks-* endpoints are a separate serving path from both pay-per-token and provisioned throughput, so they aren't gated by the system.ai EXECUTE revoke. It isn't pre-enablement behaviour that clears once t...
- 4 kudos
- 83 Views
- 2 replies
- 2 kudos
Resolved! ConcurrentAppendException on parallel MERGE into the same Delta table, despite disjoint keys
Hi everyone, Hoping someone has been through this, because I am running out of ideas. Setup. DBR 15.4 LTS, Unity Catalog managed Delta table, roughly 900 million rows. Six upstream sources each write into the same target table through their own job, ...
- 83 Views
- 2 replies
- 2 kudos
Latest Reply
- 2 kudos
Hi @Islam_hoti This is a classic "Optimistic Concurrency Control" bottleneck in Delta Lake. Even with partitions, the MERGE operation requires a metadata-level lock on the table's transaction log, which is where your contention is happening.Here is a...
- 2 kudos
- 426 Views
- 4 replies
- 4 kudos
Resolved! Difference between Workspace and Unity Catalog experiments when using MLflow autologging?
Hi everyone,I am trying to understand the exact differences between using Workspace experiments versus Unity Catalog experiments, specifically in the context of MLflow autologging (mlflow.autolog()).Does autologging behave differently depending on wh...
- 426 Views
- 4 replies
- 4 kudos
Latest Reply
- 4 kudos
Great question! I’d also be interested to know if autologging has any behavioral or permission differences with Unity Catalog experiments, especially around governance and model lineage.
- 4 kudos
- 958 Views
- 2 replies
- 1 kudos
Data practitioner in AI Era
As the AI revolution takes off in 2025, there is a renewed emphasis on adopting a Data-First approach. Organizations are increasingly recognizing the need to establish a robust data foundation while preparing a skilled fleet of Data Engineers to tack...
- 958 Views
- 2 replies
- 1 kudos
Latest Reply
- 1 kudos
Absolutely agree! AI is only as powerful as the data behind it. The evolving role of data engineers in building that foundation is definitely worth discussing.
- 1 kudos
- 82 Views
- 2 replies
- 4 kudos
Resolved! AUTO CDC FROM SNAPSHOT schema evolution in history tracking columns
Does the create_auto_cdc_from_snapshot_flow() allow for expansion of included columns in track_history_column_list or track_history_except_column_list without doing a full refresh? My intended flow is: Bronze - Managed ingestion of table (scd 1) usin...
- 82 Views
- 2 replies
- 4 kudos
Latest Reply
- 4 kudos
Adding a column and changing the history-tracking configuration need separate checks. Databricks describes adding columns as generally safe without a full refresh:"Plan your schema to avoid changes that require a full refresh. Adding columns is gener...
- 4 kudos
- 64 Views
- 2 replies
- 2 kudos
Slow Running SQL Query
The following Query is running slowly even though it returns only ~ a million rows. It takes around 25 mins. I am trying Produce a pre-aggregated summary table (grouped by 9 dimension attributes) from a large fact table, to be consumed by a Power BI ...
- 64 Views
- 2 replies
- 2 kudos
Latest Reply
- 2 kudos
With 1.4B rows, I’d first check join cardinality and whether filtering happens before the joins. Early reduction of the fact table could make a huge difference, especially before the final GROUP BY.
- 2 kudos
- 124 Views
- 4 replies
- 2 kudos
Compatibility Mode for AWS Redshift
Hello,I have several tables loaded from S3 using Lakeflow pipelines which generates streaming table and materialized views. Want to be able to read these tables in AWS redshift.Based on the Databricks documentation for compatibility mode:Databricks C...
- 124 Views
- 4 replies
- 2 kudos
Latest Reply
- 2 kudos
@HZ your understanding of Spectrum's Delta reader is correct: it requires manifests. The alternative is Redshift's Iceberg access through Glue catalog federation, which AWS documents for Unity Catalog. These are separate integration paths.For your La...
- 2 kudos
- 108 Views
- 3 replies
- 2 kudos
Failed to resolve External API
I am currently testing on a Premium trial, but my workspace is unable to resolve my external API to pull data. May I know if I need to update anything on the settings side to fix this issue?Thanks!
- 108 Views
- 3 replies
- 2 kudos
Latest Reply
- 2 kudos
This is usually related to network access rather than the Premium trial itself. I’d check the workspace’s serverless/external network access settings and make sure the API domain is allowed, then verify DNS resolution and any firewall or IP allowlist...
- 2 kudos
- 129 Views
- 2 replies
- 1 kudos
Dark mode setting ignored when embedding BI dashboard inside Databricks App
When using the '@databricks/aibi-client' library to embed a BI dashboard inside a databricks app, the " colorScheme: 'dark' " setting is ignored when the app's 'prefer-color-scheme' is 'light' by default.As a result, I am unable to set my dashboards ...
- 129 Views
- 2 replies
- 1 kudos
Latest Reply
- 1 kudos
Hi @DB-RKL Yes I have followed the example in the AIBI client docs.I have verified that the colour scheme will render in dark mode just fine when running my app (Nuxt.js framework) in a local server, but when running it inside a Databricks App server...
- 1 kudos
- 42 Views
- 0 replies
- 0 kudos
Are attached-but-idle serverless notebooks billed? Idle timeout? 32 min job, 86 min billed
Hi all, hope everyone is doing well. Two questions I am hoping someone can help with: is there a way to set an idle timeout for serverless notebooks, and is an attached-but-idle serverless notebook billed while it sits there?Some background on why I ...
- 42 Views
- 0 replies
- 0 kudos
- 204 Views
- 4 replies
- 0 kudos
AI/BI Dashboards
Hi AllMy client is looking to create a few dashboards in DataBricks instead of Power BI. We already have a template in Power BI (header for each page, information about out BI Page) in each of our power bi report which has colors, fonts, images etc.H...
- 204 Views
- 4 replies
- 0 kudos
Latest Reply
- 0 kudos
Hi @km1837, Databricks AI/BI Dashboards have solid theming and organisational capabilities that should cover most of what you are used to in Power BI. AI/BI Dashboards don't have a "template file" concept the way Power BI does with .pbit files, but y...
- 0 kudos
- 1653 Views
- 4 replies
- 2 kudos
External embedding for reports using federated credentials fails
Hi,We are implementing external dashboard embedding in Azure Databricks and want to avoid using client secrets by leveraging **Azure Managed Identity** with **OAuth token federation** for generating the embedded report token.Following OAuth token fed...
- 1653 Views
- 4 replies
- 2 kudos
Latest Reply
- 2 kudos
Databricks still hasn't committed to addressing this issue.The only workaround is to get enrolled in the private preview for Dashboard Embedding with SSO (Embed with SSO Preview User Guide - Google Docs). This only works if you're already authorizing...
- 2 kudos
| User | Count |
|---|---|
| 1838 | |
| 885 | |
| 809 | |
| 472 | |
| 313 |
