Warehousing & Analytics topics

NetSuite connector (Lakeflow Connect) — transactionline refresh takes 3.5–4.8 hrs regardless

tcranoratmcgee — Wed, 05 Aug 2026 15:09:23 GMT

Setup: Lakeflow Connect Netsuite pipeline (UC Connection, deployed via UI), Azure Databricks. Target table is netsuite.transactionline, currently about 13.8M rows, 395 columns, CDC ingestion type (I know I know but there aren't that many rows changing per day and I want that history)

Symptom: every refresh of this table takes 3.5 to 4.8 hours regardless of how much data actually changed

Examples:

Run Date	Source Rows (insert + Update)	executionTimeMs	materializedSourceTimeMs	scanTimeMs	rewriteTimeMs
2026-07-24 (initial backfill)	13,678,242	13,458,451 (3.7h)	13,374,368	24,581	59,462
2026-07-30	76,634	12,662,653 (3.52h)	12,644,040	2,864	15,735
2026-07-31	13,016	17,162,963 (4.77h)	17,150, 004	2,458	10,489
2026-08-01	11,869	12,739,450 (3.54h)	12,726,862	2,200	10,376
2026-08-03	8,257	13,023,688 (3.62h)	13,010,720	3,168	9,788
2026-08-04	21,030	13,562,017	13,547,921	2,450	11,634

The actual Delta merge/write mechanics are consistently seconds. Essentially 100% of executionTimeMs is materializeSourceImtMs and that number is flat at 3.5+ hours whether the run is picking up 8k changed rows or the full 13.68M initial backfill

Question: Sice runtime isn't scaling with rows actually written,, it looks like the connector's extraction step is doing a full scan on all the Netsuite transactionlines to comute the incremental diff rather than using a standard "modified since" pull. Has anyone else noticed this behavior? Is there a way to configure or tune the way this connector filters its source query? Or is this expected behavior for this table given limitations on Netsuite's end?

Happy to share pipeline config or more output history if useful for troubleshooting.

Apps proxy returning 502 despite healthy app runtime

julieglass — Tue, 04 Aug 2026 18:43:58 GMT

I have two Databricks Apps that show RUNNING status and clean deployment logs, but both return 502 Bad Gateway when accessing their public URLs.

Error: Both apps return "502 Bad Gateway" when accessed via their public URLs.

Troubleshooting Attempted:

- Stop/start cycle on both apps

- Created fresh app with new name and OAuth configuration

- Verified deployment logs show "Deployment successful"

- Verified Streamlit servers running on port 8080

- Both apps exhibit identical 502 behavior

Any thoughts on this would be appreciated!

How to export clean values from ai/bi dashboard table that uses html formatted columns

anmolhhns — Sat, 01 Aug 2026 13:55:20 GMT

I'm working on Databricks AI/BI dashboard containing a native visualization; to achieve custom styling such as progress bars, the dataset SQL Generates HTML strings using expressions similar to:

CONCAT( '<div style="...">', CAST(value AS STRING), '</div>' ) AS formatted_value

The HTML renders correctly within dashboard table but when the data is exported the HTML tags are also exported along with it. Is it possible to export data without HTML tags?

Visual Data Prep - Missing rows when loading tables

dax60 — Fri, 31 Jul 2026 10:02:33 GMT

Hi,

I'm experiencing an issue with Visual Data Prep where some source tables appear to contain fewer rows than expected.

When I query the same tables directly using SQL, I can see all records. However, when loading certain tables into Visual Data Prep through a Source operator, the row count appears to be much lower.

What makes this confusing is that the issue only happens for some tables.

For other tables, when I set Rows scanned = Max, all rows appear to be available as expected.

For the affected tables, the lower row count is already visible immediately after the Source step, before I perform any joins, filters, or transformations.

I've included two screenshots:

A SQL count showing the table contains 4,168 rows.
The same table loaded into Visual Data Prep, where the Source operator only shows approximately 2,692 rows.

This difference is visible before any transformations are applied.

My questions are:

Does Visual Data Prep automatically sample data for certain tables?
Can "Truncated data" affect the displayed row count?
Is there any known limitation that would cause some source tables to show fewer rows than are available in SQL?
Has anyone experienced different row counts between a SQL query and the Source operator in Visual Data Prep?

Any suggestions on how to troubleshoot this would be greatly appreciated.

Thanks!

Can {{input}} from a For Each task be used as a dashboard subscription destination_id

dario66952 — Tue, 28 Jul 2026 15:53:32 GMT

Hello everyone,

I am using a DAB to send a dashboard snapshots to notification destinations.

The intended design is to use a For each task for multiple team/destination pairs. As a minimal test, I am iterating over an array containing one notification destination ID:

variables:
  destination_1_nonprod:
    lookup:
      notification_destination: destination_1_nonprod

  team_subscriptions:
    description: JSON array of notification destination IDs
    default: '["${var.destination_1_nonprod}"]'

resources:
  jobs:
    job1:
      ...

      parameters:
        - name: team_subscriptions
          default: ${var.team_subscriptions}

      tasks:
        - task_key: send_team_reports
          for_each_task:
            inputs: "{{job.parameters.team_subscriptions}}"
            concurrency: 5

            task:
              task_key: send_team_report
              dashboard_task:
                dashboard_id: ${resources.dashboards.my_dashboard.id}
                warehouse_id: ${var.warehouse_id}
                subscription:
                  subscribers:
                    - destination_id: "{{input}}"

The bundle correctly resolves team_subscriptions to a JSON array containing the notification destination UUID and I can see that in the UI inside the task "job parameters".

The job succeeds but no notification is delivered, silently ignoring the issue.

As a control test, replacing the dynamic reference with the bundle lookup works correctly:

subscription:
  subscribers:
    - destination_id: ${var.destination_1_nonprod}

I also

I also considered using a json object with key:value and iterating over it using:

destination_id: "{{input.destination_id}}"

but it still silently fail the delivery

Does dashboard_task.subscription.subscribers[].destination_id support Lakeflow dynamic value references such as {{input}} or {{input.destination_id}}?

If it does, what is the correct syntax or configuration?

Thank you for your attention

Data Quality Dashbaord - I have a requirement to build a custom designed quality dashboard in dbx

Databricks2206 — Thu, 23 Jul 2026 05:54:52 GMT

I have a requirement to build a custom designed quality dashboard in dbx. The requirement is like user have to view the quality of data metrics in a single dahsboard for many tables, irrespective of viewing it via inbuilt data quality option available in every table. what is the best design approcah .please explain.

AI BI Dashboard Table filtering

JZhang — Wed, 15 Jul 2026 07:31:33 GMT

Hello community of Databricks

I need some help over a specific issue in AI/BI Dashboard filtering

My current dashboard has 2 widgets:

- A map containing the shopID and it's current GPS location

- A table showing all the available shopID with some metadatas

I was wondering if it is possible to click on a specific row of the table to filter the specific shop in the map, but right now it returns me "Unable to render visualization" when clicking on a specific row of the table.

The filter widget is available but i would not say it is the most intuitive way since i can't even copy the shopID from the table (the default is selecting a specific row)

Let me know if additional infos are required to solve my doubt!

Is it possible to configure the evaluation runs overview?

lkt1 — Tue, 14 Jul 2026 08:06:15 GMT

I would like to be able to add columns to this overview, if possible. For example, the latency of the model is available, when I click on a run, but I would like to see the average latency, similarly to, how the score is shown. Of course, I could add a new score, which is latency, but I'd like to know if I can use the already existing data.

Along with this, I would like to be able to configure it such, that I can see the percentiles and medians of my score. Is any of this possible to do in the overview?

Genie One – Unable to Extract Usage Details

devyani_k — Tue, 14 Jul 2026 02:52:54 GMT

We are looking to create a Genie usage report covering natural language querying activity across three channels — the Genie One web app, Genie One Teams app, and Genie One mobile app. The report is intended to surface insights around user activity, question types, query patterns, feedback, and session behaviour.

What we are currently able to extract via system tables (across all channels)

- User email
- Number of questions asked
- Source breakdown by channel (Web App / Teams / Mobile) with question counts
- First and last active date
- Feedback count

What we are unable to extract

For questions asked across all three channels (web app, Teams, and mobile), we are currently unable to retrieve:

- Actual question text asked by the user
- SQL queries executed in response to the question
- Tables and assets queried
- Feedback type (thumbs up / thumbs down) and associated feedback text where provided

Questions

1. Is there a system table that currently stores this information, or a mapping that would allow us to extract it?
2. If not, is there an alternative approach recommended by Databricks to access this data?

Dashboard Creation Requiring Workspace Access Entitlement?

mkjun — Wed, 08 Jul 2026 20:41:36 GMT

Hi Team,

I have analysts set up in Databricks with the "Databricks SQL access" entitlement who are no longer able to create a new Dashboard on their own and just getting a blank page when they try. The API response to their create dashboard request is: "User does not have the workspace-access entitlement", and it seems that only users with the "Workspace access" entitlement are still able to create a new dashboard.

Is this an intentional change to Databricks entitlements? The latest documentation on Databricks entitlements makes no mention of this.

If this is intentional, I would strongly disagree with this change. These are analysts that primarily focus on report creation and ad-hoc queries; they shouldn't have to be granted all the capabilities that come with the "Workspace access" entitlement if they're just making Dashboards.

Thank you,
Matthew

Dashboard Variable Display name

amekojc — Tue, 07 Jul 2026 14:57:40 GMT

In databricks documentation it says that there's a display name when creating dashboard variables but I'm not seeing it in the dashboard settings. Does anyone know where to find them?

Asset Bundles Setup

Cristian17 — Wed, 01 Jul 2026 13:27:42 GMT

Hi everyone,

I have a question about the best practice for organizing Databricks Asset Bundles.

I have multiple projects, and each project has its own notebooks and jobs.

Is it recommended to:

create a single bundle that includes all projects, or
create a separate bundle for each project?

Also, my understanding is that the Databricks UI doesn't support deploying only selected resources from a bundle, while the Databricks CLI allows this (for example, using bundle deploy --select).

Does this limitation influence how you structure your bundles? What approach has worked best for your team?

Network time-out

GJL-Geophysics — Mon, 22 Jun 2026 10:22:16 GMT

When trying to download the public domain seismic data of Equinor’s Volve field (https://www.equinor.com/energy/volve-data-sharing) using a browser (Edge or Chrome) this fails after 2 hours due to network errors. Resuming only provides once off relieve until it fails again. Laptop, Lenivo running Win11, settings are such to keep system active (never sleep). Network speed is 600+ Mbps download and 100+ Mbps upload. Browser downloads only at 8-9 Mbps. With 2.1 and 1.2 Tb for the two files, download time is initially estimated between 2-5 days!

Any suggestions how to resolve?

Is there an FTP option with faster downloads (perhaps not for personal free databricks account)?

AI/BI Dashboard Navigation: Nested Tabs, Page Structure, and Reusable Visuals

sheeetal — Tue, 16 Jun 2026 06:53:46 GMT

I have a few questions about AI/BI Dashboards:

Is there a way to create nested tabs (tabs within tabs), or are dashboards limited to a single level of page navigation?
For dashboards with many visualizations, what is the recommended approach to improve navigation and avoid excessive scrolling? Need hyperlinking solutions
Can users jump directly to a specific section/page within a dashboard?
Can visualizations/widgets be copied and reused across different dashboards, and are there any limitations to doing so?

I'd appreciate any guidance, best practices, or documentation references.

Does combo chart have group by color function ?

emily02 — Thu, 11 Jun 2026 19:29:51 GMT

Does combo chart have group by color function by specific field (type)?
Like

Line Chart: the line will perfectly split into multiple lines of different colors, with each line representing a distinct Type.

Bar Chart: The original single bar will also automatically transform into a Stacked Bar chart using different colored blocks to represent for each Type.
like what we have in tableau

How to show percent of total that recognizes global filter selections

amekojc — Thu, 04 Jun 2026 20:54:26 GMT

I'm trying to implement a percent of total measure in pivot table. I succeeded in creating a measure using the below formula.

However, I want this measure to recognize global filter selections. For example, I want to show emissions share by category. But if I remove 1 or 2 values from category global filter, I'd like the total share to recalculate the share with the remaining categories.

Was anyone able to achieve this?

How to avoid excel automatic data formatting when loading from excel plugin

weerasa — Wed, 03 Jun 2026 22:36:30 GMT

Excel suddenly started automatically formatting data (convert everything to text) when loading from the new Azure Databricks Excel plug-in. I can still see my default formatting in the Databricks sideload. Just wondering if this behaviour was caused by some recent Excel update or Azure Databricks plug-in update.

Embedding a dashboard in Databricks

vvanag — Thu, 28 May 2026 14:47:06 GMT

Hi,

I try to embed my dashboard in Databricks Notebook as

some_html = """ <iframe src="...." width="100%" height="600" frameborder="0"> </iframe> """ displayHTML(some_html)

then I get an error of the form:

This dashboard is unavailable

Please try again later. Show details

And the details says:

"""

RPC has already been initialized in the current window. To minimize the opportunities for incompatible changes, only one RPC host can be initialized inside a global context

"""

How can I embed the dashboard? The above is not making sense to me.

Problem rendering HTML in IPywidgets

vvanag — Thu, 28 May 2026 07:38:03 GMT

The below code works fine in Colab/VSCodium/JupyterNotebook

import ipywidgets as widgets from IPython.display import display, HTML output = widgets.Output() with output: display(HTML(some_html)) output

But it does not work in Databricks

testme.html

Databricks excel plugin error

oyslan — Fri, 22 May 2026 10:57:55 GMT

Hi, i have an error with the excel pluging.

It was working fine until today. I can login and browse the catalog but I get an error when trying to query any data. It seems like its sending in a blank warehouse id. I have choosen a warehouse and its running.

Anyone who knows what the reason can be?

Error message:
Excel operation failed in TableSelector: {"ok":false,"status":400,"statusText":"","json":{"error_code":"INVALID_PARAMETER_VALUE","message":"INVALID_PARAMETER_VALUE: is not a valid endpoint id."},"body":{},"statement":"SELECT `report_concept_id`, `code`, `name_no`, `name_en` FROM `tower-cs_dev_base`.`heartbeat`.`report_concept` LIMIT 1000","warehouseId":""}