All Warehousing & Analytics posts

Re: Facing Data Truncation Issues in Databricks Dashboards

Prashant_Mishra — Thu, 24 Sep 2026 06:41:18 GMT

@Akshay_Petkar is probably talking about the limitation of pivot table data export rows. This is limited to 10k and not 1k. So, changing 1k setting will not help here. The underlying sql query returns 10k+ rows, but the pivot table UI truncates rows more than 10k. This is Databricks product feature. Is there a work around to get more then 10k records exported from pivot data?

Re: "Revoke" permissions for SQL-Warehouse with API

ashford — Tue, 22 Sep 2026 08:30:44 GMT

You may want to check whether the API expects a specific permission type for revocation rather than a special value such as NO_PERMISSIONS or REVOKE. The error suggests the request is reaching the endpoint, but the permission payload isn't matching the accepted schema.

I’d compare the revoke request against the API documentation/example for the exact SQL Warehouse permission type and payload format. Also check whether permissions are inherited from a catalog, schema, or group, since those may not be removable directly from the warehouse.

Re: ODBC v2.12- 503 "no Retry-After header"

saf_dbx — Mon, 21 Sep 2026 09:09:20 GMT

Thanks I will try setting the parameter and see if that will help. The databricks SQL Warehouse is already serverless in my case.

Re: ODBC v2.12- 503 "no Retry-After header"

ThomazNeto — Sun, 20 Sep 2026 11:03:08 GMT

Hi,

I've hit this exact error in a few environments, so maybe this helps.

First, the 503 itself is coming from the Databricks side, not from your BI tool or the driver. In every case I've traced it, the timestamps lined up with the SQL Warehouse being in a transitional state: auto-stop / cold start, scaling up or down, or a node being replaced. The service answers 503 for a few seconds and then recovers on its own. Worth cross-checking your failure times against the warehouse events and Query History.

What makes it fatal is the driver behavior. Out of the box, the ODBC driver only retries a 503 when the server includes a Retry-After header, and Databricks doesn't always send one. Without the header the driver gives up immediately and you get the Hardy (124) error.

About the parameter: EnableRetryWithoutRetryAfterHeader does exist, but you won't find it in the Databricks docs. I couldn't find it in the docs.databricks.com ODBC pages or in the release notes from 2.9.1 up to 2.12.1. The only place I've seen it written down is a Qlik support article from May this year, which says it's available from driver 2.9.1 onward and defaults to 0. It's one of those "key name only" options that don't show up in the DSN dialog.

To set it:

Windows: open regedit, navigate to HKEY_LOCAL_MACHINE, then SOFTWARE, then ODBC, then ODBC.INI, then your DSN name, and add a string value named EnableRetryWithoutRetryAfterHeader with value 1. Or append it to the connection string if your BI tool lets you.

Linux/macOS: add the line EnableRetryWithoutRetryAfterHeader=1 to the DSN section in odbc.ini.

DSN-less connection string: add EnableRetryWithoutRetryAfterHeader=1 as one more semicolon-separated property.

You're on 2.12.0, so version isn't a problem.

On the warehouse side, if this happens a lot, look at the auto-stop setting (extend it or disable it during the refresh window), or move to Serverless, which starts in seconds and cuts down on the cold-start 503s considerably. If you want proof for a support ticket, turn on driver logging (LogLevel=4 plus LogPath) and you should see the raw 503 with no Retry-After in the trace.

One caveat: since the parameter isn't officially documented by Databricks, I'd test it in your environment before rolling it out broadly. It's been stable where I've used it, but I can't point you to an official Databricks reference for it.

Hope this helps.

ODBC v2.12- 503 "no Retry-After header"

saf_dbx — Sun, 20 Sep 2026 08:08:26 GMT

I'm connecting to a Databricks catalog via a SQL Warehouse using the Databricks ODBC Driver (v2.12.0). During one of the fetches from my BI application, the transaction aborts with:

Database operation [SQLExecute] failed with error [[Databricks][Hardy] (124) A 503 response was returned but no Retry-After header was provided. Original error: Unknown]

I've seen references to a driver parameter called `EnableRetryWithoutRetryAfterHeader` that's supposed to make the driver retry a 503 even without a `Retry-After` header, but I can't find this documented in official Databricks ODBC docs.

This error doesn't come all the time, and most of the time it works okay.

Can anyone confirm:
If such a parameter exists, how can one configure this on the DSN settings?

Thanks in advance for any pointers.

Re: UC Delta API: managed-table READ_WRITE credentials rejected despite explicit User-Agent

GabFernandes — Tue, 15 Sep 2026 14:15:13 GMT

Hi @ablanchard .

The behavior you're seeing is expected for the Delta protocol-level credential endpoint (/api/2.1/unity-catalog/delta/v1/.../credentials). A few things to note:

Managed Delta table writes from external Delta clients are in Public Preview (docs)During this preview, the Delta protocol API validates the User-Agent header for READ_WRITE operations to gate access to recognized/registered connectors. This is why READ succeeds (it's GA) while READ_WRITE fails.
Consider using the general credential vending API instead. The documented public endpoint is POST /api/2.1/unity-catalog/temporary-table-credentials with a JSON body containing table_id(the table's UUID) and operation: "READ_WRITE". This is the officially documented path for external engines and may not enforce the same User-Agent gating. You can get the table UUID via the ListTables API with include_manifest_capabilities enabled — look for tables marked HAS_DIRECT_EXTERNAL_ENGINE_WRITE_SUPPORT.
Prerequisites: the metastore must have external_access_enabled = true, and the principal needs EXTERNAL_USE_SCHEMA on the parent schema. For catalog commits specifically, the table property delta.feature.catalogManaged must be set to 'supported'.
For the Delta protocol path specifically, connector registration/onboarding with Databricks is likely required during the Public Preview phase. Filing a feature request via the Databricks Ideas portal or opening a GitHub issue on the unitycatalog repo would be the best way to request formal onboarding for HarborSQL.
If my answer was helpful, please consider marking it as accepted solution!

Re: External embedding for reports using federated credentials fails

iamgoce — Mon, 14 Sep 2026 19:43:45 GMT

Databricks still hasn't committed to addressing this issue.

The only workaround is to get enrolled in the private preview for Dashboard Embedding with SSO (Embed with SSO Preview User Guide - Google Docs). This only works if you're already authorizing users with the same IdP SSO (e.g., Entra ID) and the users have the adequate access to the workspace, dashboard and SQL warehouse.

But for embedding dashboards for public consumption (non-Databricks users), the above won't work. One has to use a SP with a client secret.

UC Delta API: managed-table READ_WRITE credentials rejected despite explicit User-Agent

ablanchard — Sat, 12 Sep 2026 23:01:29 GMT

Hi everyone,

I’m developing HarborSQL, an open-source SQL engine using Apache DataFusion and delta-rs. I’m working on managed Delta table writes through the UC Delta API with catalog commits.

Using the same workspace, managed table, and user identity:

- Table metadata GET succeeds (200).
- Credentials GET with operation=READ succeeds (200).
- Credentials GET with operation=READ_WRITE fails (400).

This happens with both PAT and browser-based OAuth U2M authentication.

The failing request is:

GET /api/2.1/unity-catalog/delta/v1/catalogs/{catalog}/schemas/{schema}/tables/{table}/credentials?operation=READ_WRITE
User-Agent: HarborSQL_HarborSQL/0.1.9
Accept: application/json
Content-Type: application/json

The response says:

The provided User-Agent 'HarborSQL_HarborSQL/0.1.9' is insufficient.

It also directs connector developers to contact Databricks support, but I don’t have access to support.

The header follows the documented partner User-Agent format. Using harborsql/0.1.9 produces the same error. The failure occurs during credential vending, before any data write or commit request.

Does managed-table write access require connector registration, allowlisting, or additional preview enablement? Are there client-identification requirements beyond the documented User-Agent format?

If onboarding is required for independent open-source engines implementing catalog commits, could someone point me to the appropriate process or team?

Thanks!

Re: Controlling allow/ask/deny lists for predelivered MCP connectors

DB-RKL — Fri, 11 Sep 2026 14:21:14 GMT

Hey @Leembo

system.ai.microsoft_365 is now a 1st class UC object ("MCP service") that runs through Unity Gateway, same as the external connectors you register yourself.

Two levels of control:

- Whole connector on/off: an EXECUTE grant, so you can allow/deny M365 per user/group centrally today (GA).
- Denying specific tools tenant-wide w/ service policies: guardrails you attach to the MCP service to allow/deny/require-approval on individual tool calls. Service Policies are what enforce a per-tool denylist centrally so the per-user toggles can only pick within what you've allowed up top. (Beta)

I hope this helps

Controlling allow/ask/deny lists for predelivered MCP connectors

Leembo — Fri, 11 Sep 2026 10:03:27 GMT

Hello,

We are looking at utilizing the pre-delivered MCP connector to M365 via Genie One. The connector works great, but each user can individually configure the allowable tool calls without any central governance layer.

Are we missing something or is that currently a gap in pre-delivered connectors? Ideally, as an account administrator I would be able to select which tools are completely denied at a tenant level for users as a guardrail.

I know you can do that for externally registered MCP connectors by managing it at a catalog level, is that going to be possible for predelivered MCP connectors as well?

Thanks,
Linas M.

Re: Optimizing B2B E-Commerce & Construction Supply Chain Pipelines (Data Architecture Case Stud

SumeshKashyap — Thu, 10 Sep 2026 16:59:03 GMT

This one's really a network/download-client issue more than a Databricks one, but since the end goal is presumably getting the data into Databricks, a couple of angles:

- Browser downloads are single-connection and easy to bottleneck -- 8-9 Mbps on a 600 Mbps line is a classic sign of a single TCP stream being capped (by the server, a proxy, or per-connection throttling), not your bandwidth. A resumable, multi-connection downloader will usually do much better: "aria2c -x16 -s16 -c <url>" (parallel connections + auto-resume), or "curl -C - -O <url>" in a retry loop for simple resume without parallelism.

- Skip the local download entirely if you can. If Equinor's Volve data is hosted on a cloud object store (worth checking their data-sharing page for an S3/Azure Blob endpoint rather than an HTTPS file server), you can often ingest it straight into Databricks via Auto Loader or dbutils.fs.cp pointed at the bucket directly -- cloud-to-cloud transfer avoids your local network entirely and sidesteps this whole problem. Worth checking the dataset's distribution page for that option before spending days downloading 3+ TB to a laptop.

If it does have to come through your local connection, aria2c with resume + parallel connections is the standard fix for the 2-hour-timeout pattern you're describing.

Re: Optimizing B2B E-Commerce & Construction Supply Chain Pipelines (Data Architecture Case Stud

SumeshKashyap — Thu, 10 Sep 2026 16:58:03 GMT

@Gharhub Good questions!!!!

A few patterns that tend to work well for this kind of catalog:

Schema for heterogeneous raw materials: rather than one wide table per material category (which breaks every time a new attribute shows up), model a core materials table (id, category, supplier, base UoM) plus a category-specific attributes column stored as VARIANT (or MAP<STRING,STRING> if you're on an older Delta version) instead of literal EAV rows. That gets you schema flexibility without a proliferation of sparse columns, while keeping core dimensions (price, supplier, category) as real typed columns for fast filtering/joins.

Real-time-ish pricing: "real-time" freight/bulk pricing is rarely true streaming end-to-end -- usually it's: ingest price/freight-rate change events via Auto Loader into a Structured Streaming pipeline, apply pricing rules as a streaming MERGE into a current_prices Delta table (keyed by SKU+region+tier), and serve reads from that table directly if consumers tolerate seconds-old data, or via an online table/Lakebase if you need sub-100ms lookups from a checkout path. Keep the pricing rules as data in a rules table joined in, not hardcoded logic, since freight/bulk tiers change often.

DLT pipeline shape: standard medallion -- Bronze via Auto Loader off raw ERP/EDI drops, Silver applying the schema above plus dedup/CDC handling (APPLY CHANGES INTO if your source gives you CDC), Gold as the pricing-and-inventory marts your BI/pricing engine reads from. For inventory specifically, use APPLY CHANGES INTO with sequencing on your source's change timestamp so out-of-order updates from multiple warehouses resolve correctly instead of last-write-wins on arrival order.

Re: Could Databricks app support the integration of Genei Room and Power BI?

balajij8 — Tue, 08 Sep 2026 16:07:08 GMT

@fjrodriguez

The integration is technically feasible. Apps built with AppKit (the recommended TypeScript + React SDK) generally include a native Genie plugin that provides conversational AI/BI capabilities with natural language queries. You can embed a Power BI report alongside this Genie interface in the same app using standard iframe embedding or the Power BI JavaScript SDK. The architecture involves a React component hosting both the AppKit GenieChat component (handles Databricks authentication automatically) and a Power BI embed container configured for Entra ID authentication. More details here

Could Databricks app support the integration of Genei Room and Power BI?

fjrodriguez — Tue, 08 Sep 2026 15:54:05 GMT

Is it possible: a Databricks App embedding Power BI ? Rather than placing Genie inside Power BI, place Power BI inside Databricks. A Databricks App (React or Streamlit) hosts native Genie chat alongside a Power BI report embedded via the JavaScript SDK.

Databricks App using the AppKit Genie plugin (GenieChat, useGenieChat, SSE streaming, conversation persistence) or the SDK directly
Power BI Embedded in embed-for-your-organization mode (user-owns-data) with Entra ID
App registration in Entra and workspace permissions in Power BI
Hosting and application lifecycle ownership

Re: Change the default font for plots

josephhawk21 — Fri, 04 Sep 2026 18:59:14 GMT

Updating every plot one by one can become a real hassle as the project grows. A global font setting is much easier for keeping the same brand style across all visual outputs, especially when using a fancy arabic font.
https://zakhrafalasmaa.com/

Re: Is it possible to embed a Genie Space inside a Power BI report?

fjrodriguez — Fri, 04 Sep 2026 11:23:10 GMT

Hi @Louis_Frolio , and what about inverting the relationship: a Databricks App embedding Power BI ? Rather than placing Genie inside Power BI, place Power BI inside Databricks. A Databricks App (React or Streamlit) hosts native Genie chat alongside a Power BI report embedded via the JavaScript SDK.

Databricks App using the AppKit Genie plugin (GenieChat, useGenieChat, SSE streaming, conversation persistence) or the SDK directly
Power BI Embedded in embed-for-your-organization mode (user-owns-data) with Entra ID
App registration in Entra and workspace permissions in Power BI
Hosting and application lifecycle ownership

Re: Is it possible to embed a Genie Space inside a Power BI report?

fjrodriguez — Thu, 03 Sep 2026 15:01:26 GMT

Thanks every one ! Was a great one :9

Re: Is it possible to embed a Genie Space inside a Power BI report?

Louis_Frolio — Thu, 03 Sep 2026 13:25:21 GMT

Hi @fjrodriguez ,

Good question, and you've already spotted the core problem. Short answer: I couldn't find any internal documentation that supports embedding a Genie Space directly inside a Power BI report visual, and the sandboxing you described makes a reliable direct embed unlikely. I'd treat that path as unconfirmed at best rather than something to build on.

Here's the nuance. Databricks does support embedding a Genie Space (the docs now call these Genie Agents) as an iframe in an external app. A workspace admin has to allowlist the embedding surface, and each end user needs explicit access to both the space and the underlying data, so users who aren't already signed in get prompted to authenticate inside the frame. That last part is exactly what a Power BI report visual can't accommodate: the sandboxed iframe blocks the cookies and redirects the Entra ID sign-in needs. A blank frame or a refusal from the Entra login page is usually that boundary, and there's nothing you can do about it from the visual's side.

So the two suggestions already in this thread are the right paths, with a few qualifications:

Power Apps visual plus the Genie Conversation API, as @Satyasai suggested. This is the closest you'll get to chat inside the report, but to be clear, it's a custom integration, not an embed of the Genie UI. You'd build a small chat front end in Power Apps, typically with a Power Automate flow behind it that calls start-conversation, polls the message status, and fetches the query result. Keep tokens and authorization in the flow or a trusted backend, never in the canvas. And think hard about the auth model: if the flow authenticates as a service principal, every report user sees data through that principal's permissions, so per-user row filtering in Unity Catalog won't apply. If per-user permissions matter, you need user-based OAuth, which adds real complexity.
A Power BI button with a Web URL action that opens the Genie Space in a new tab, as @balajij8 suggested. Simplest option, fully supported, complete Genie experience, no nested login problem. The cost is that users leave the report.

And if the underlying requirement is interactive analytics with natural language rather than "must stay in Power BI," take a gander at Databricks AI/BI dashboards, which @balajij8 also mentioned. Genie is natively integrated there, no embedding gymnastics required.

My honest recommendation: start with the button to prove out demand, and invest in the Power Apps/API approach only if chat inside the canvas turns out to be a hard requirement.

References:

Embed a Genie Agent in an external app: https://docs.databricks.com/aws/en/genie/embed
Manage embedding and allowed destinations: https://docs.databricks.com/aws/en/ai-bi/admin/embed
Genie Conversation API: https://docs.databricks.com/aws/en/genie/conversation-api

Hope this helps.

Regards, Louis

Re: Is it possible to embed a Genie Space inside a Power BI report?

balajij8 — Thu, 03 Sep 2026 10:26:17 GMT

@fjrodriguezEmbedding an auth gated Databricks Genie Space directly into a Power BI report visual generally runs into nested authentication and sandboxing constraints. You can check Databricks AI/BI dashboards instead of Power BI if interactive analytics is required. AIBI dashboards has a significantly better Databricks integration (tables, genie spaces) than other BI tools. You can use a standard Power BI button with a Web URL action that launches the Genie Space in a new tab if feasible as it guarantees full Genie conversational functionality and avoids iframe authentication failures.

Re: Is it possible to embed a Genie Space inside a Power BI report?

Satyasai — Thu, 03 Sep 2026 10:02:32 GMT

Hi @fjrodriguez

you can try this,

build a lightweight chat interface inside the Power Apps visual that communicates with Databricks asynchronously via APIs and embed this power Apps in Power BI.