All Warehousing & Analytics posts

Re: Dashboard DAB Deployment: default_catalog and default_schema do not work for metric views anymor

Louis_Frolio — Fri, 02 Oct 2026 17:10:38 GMT

Hello @Capri, I took a look at both internal and external documentation and here is what I found.

You're not alone, and nothing on your side caused this. @SamuelHarris reported the identical failure on Sep 30 on this thread (https://community.databricks.com/t5/warehousing-analytics/recommended-local-development-workflow-for-dashboard-ci-cd-with/td-p/156465), same invalid asset name 400 and the same invalid dependency "${var.catalog}" error when he tried the variable workaround. The failing call is the server side POST /api/2.0/lakeview/dashboards, so your CLI version isn't the culprit. Something changed in the Lakeview API's validation on or around Sep 29.

The pattern you're using is the documented one. The bundle docs describe dataset_catalog and dataset_schema as the defaults for every dataset in the dashboard unless the query says otherwise, and in May a Databricks employee (stbjelcevic) confirmed on that other thread that the deploy-target .lvdash.json should hold unqualified asset_name values and that there's no way to parameterize asset_name inside the JSON body. So the API is now rejecting a shape the docs still call valid. I can't tell from public documentation whether that's intentional or a regression, so please open a support ticket with your minimal repro, CLI version, cloud, the exact bundle deploy output, and links to both threads. An issue on the CLI repo wouldn't hurt either, since that team owns the dataset_catalog/dataset_schema fields on the bundle side.

Credit to @Satyasai for the diagnosis: validation on asset_name got stricter, and Fix 1 (fully qualify it) does work. Fix 2 is where I'd steer you differently. The .tmpl convention is only processed by databricks bundle init when it scaffolds a new project, using Go template syntax. At bundle deploy the CLI reads your .lvdash.json as-is and ships it to the API as serialized_dashboard. Bundle variable substitution covers the YAML, not that JSON body, so when the CLI spots ${var.catalog_name} in there it treats it as a reference it can't resolve, and that's your no such node "" error. Your variables belong in the resource YAML, which is what you already have. One side note for anyone copying the published dashboard example: it declares catalog_name/schema_name but references ${var.catalog}/${var.schema} in the resource block. Those names have to match.

What works today while keeping dev and prod separate: render the JSON in your pipeline before deploy, which is the pattern stbjelcevic recommended in May even before the breakage.

# dashboard.template.json contains: "asset_name": "__CATALOG__.__SCHEMA__.test_metric_view"
sed -e "s/__CATALOG__/${CATALOG}/g" -e "s/__SCHEMA__/${SCHEMA}/g" \
    dashboard.template.json > dashboard.lvdash.json
databricks bundle deploy -t prod

(Those are plain shell variables from your pipeline, not bundle variables.) If you'd rather stay pure bundle, keep one fully qualified JSON per environment and override file_path in each target; it costs you duplicated JSON but skips the transform step. Either way, keep dataset_catalog and dataset_schema on the resource for SQL-based datasets, since the new validation only fires on asset_name.

If you need something in place right now and don't need the native metric view dataset experience, a stopgap is a SQL dataset against the metric view, something like SELECT dim, MEASURE(my_measure) FROM test_metric_view GROUP BY dim. SQL datasets resolve through the defaults at query time rather than being validated at create time. I haven't tested that against the new validation, so treat it as a lead rather than a guarantee.

At the end of the day, the transform step is a few lines of shell and was the recommended workflow anyway, so I'd lean on that and let the ticket run its course.

References

Related thread with the May confirmation and the Sep 30 report: https://community.databricks.com/t5/warehousing-analytics/recommended-local-development-workflow-for-dashboard-ci-cd-with/td-p/156465
Bundle dashboard resource fields: https://docs.databricks.com/aws/en/dev-tools/bundles/resources
Bundle example for dashboard catalog and schema parameterization: https://docs.databricks.com/aws/en/dev-tools/bundles/examples
Bundle variables and substitutions: https://docs.databricks.com/aws/en/dev-tools/bundles/variables
Bundle templates (.tmpl and bundle init😞 https://docs.databricks.com/aws/en/dev-tools/bundles/templates
Lakeview create dashboard API (dataset_catalog/dataset_schema😞 https://docs.databricks.com/api/lakeview/v1/create-dashboard
Databricks CLI issues: https://github.com/databricks/cli/issues

Regards, Louis.

Re: Unable to connect to Tableau Cloud

datarich — Wed, 30 Sep 2026 13:51:19 GMT

It looks like that error comes from Tableau Cloud rather than Databricks, that documentation guide should be the right one.

There isn't an "Explore in Tableau" setting to turn on in Tableau Cloud. Web authoring is always on for Cloud (https://help.tableau.com/current/online/en-us/web_author_enable.htm). So it's usually the second part of the message, the user's role.

Worth checking:

The account you sign in with during the flow has a Creator or Explorer (can publish) site role, or Site Administrator Creator/Explorer. Not Viewer.
You're landing on the right site after sign-in, if your login has access to more than one.
The Default project (or wherever the draft opens) allows Web Edit and Save for your group.

If all of that looks right and you still get the error, I'd raise it with Tableau support, as it may need checking on their side.

How to build FinOps chargeback by cost centre — allocating the whole cloud bill, not just your DBUs

Valeria_Koz_DBX — Wed, 30 Sep 2026 13:46:15 GMT

Every IT function I've worked with eventually asks the same question, and can never answer it cleanly and clearly: "What did each team actually cost us last month?"

When speaking with the platform team at a large UK bank, the realisation was that the blocker wasn't that the data was missing, or even scattered. It was all sitting right there, in one big pile. Every cost row was captured. The problem was that none of the costs were marked. Nothing on a row told you which team it belonged to — so you could see the total to the penny and still have no idea who to hand it to. A platform function lives or dies on being able to show teams what they cost, and this one was staring at the whole bill unable to split it.

So we fixed it — and the fix wasn't more data, it was a name against every row. Here's the pattern, and it lives entirely in SQL over governed tables.

The principle: a cost you can't attribute is a cost nobody owns

Here is the step-by-step of how we did it:

The work splits into two stages: first allocate the whole bill across every cost type, then go inside the Databricks spend and attribute it to the pipeline, dashboard or team that caused it. A note upfront: nothing here is cloud-specific — the same pattern works whether your bill comes from AWS (CUR), Azure (Cost Management) or GCP (billing export). Normalise them all to a common shape (FOCUS is the emerging standard) and the rest is identical.

Step 1: Land all cost sources in one governed table. Pull your cloud provider's billing export and the Databricks billing/usage data into one schema, one grain (one row per resource per day). Keep a cost_type column so you can always see the split: Storage · Compute (VMs) · Databricks (DBUs) · Application / platform services.
Step 2: Agree one tagging taxonomy across all of it. Decide the allocation key (team / cost centre) once and apply it to every cost type — cloud resources and Databricks workloads. Enforce it going forward with cloud tag policies and Databricks compute/budget policies so the tag lands in billing automatically.
Step 3: Attribute every row to a team. Prefer the tag; fall back to a resource→owner mapping; label the rest honestly.
1. SELECT cost_type,
  coalesce(tags.team, owner_map.team, 'UNALLOCATED') AS team,
  sum(cost) AS cost
  FROM finops.all_cloud_costs c
  LEFT JOIN finops.owner_map ON c.resource_id = owner_map.resource_id
  GROUP BY 1, 2;
Step 4: Roll up cost-type × team. Now you can tell any team its full footprint — "here's your Storage, your VMs, your Databricks, your app services" — not just one slice.

Stage 2 — go inside the DBU spend

"Databricks: £X" isn't actionable. Within it, cost is driven by three different things, and each needs its own tag.

5. Step 5: Tag every unit of Databricks work. Put a team/owner tag on each pipeline/job, each SQL warehouse, and each dashboard/app. Job and warehouse tags flow into the usage records as custom tags — that's your primary key for pipeline and job DBUs.

6. Step 6: Attribute shared-warehouse DBUs to the actual consumer. Dashboards and ad-hoc/department queries usually share a warehouse, so tags alone won't split them. Join the warehouse's usage to query history and apportion its DBUs by who actually ran what.
-- split a shared warehouse's DBUs across consumers by compute used
SELECT q.executed_by, q.query_source, -- dashboard / user / dept
sum(q.execution_secs) / sum(sum(q.execution_secs)) OVER () AS share
FROM system.query.history q
WHERE q.warehouse_id = :wh
GROUP BY 1, 2;
Multiply each consumer's share by the warehouse's DBU cost for the period.

7. Step 7. Produce the two-level view. Team → (Storage / VM / DBU / App), and within DBU → (which pipelines, which dashboards, which departments' queries). That's the level at which a team can actually change its bill.

The hard part — shared pipelines and data used by many teams

The moment one pipeline ingests data that several teams consume, "who pays?" has no obvious answer. Don't guess — pick a model and state it.

8. Step 8. Find the shared assets. Use lineage to spot any pipeline/table with more than one downstream consumer. Those are the rows you can't simply tag to one team.

9. Step 9. Choose an allocation basis and make it explicit. Common options, roughly fairest first:

Usage-driven cross-charge — split the pipeline's cost across consuming teams by how much they actually use the output (bytes/rows read, or query compute against the downstream tables, from lineage + query history). Fairest, and it rewards teams that consume less.
Equal split — divide across named consumers. Simple, defensible when usage is hard to measure.
Platform-funded shared service — the platform/EDP team owns it centrally as common infrastructure, funded by leadership, not cross-charged. Right for truly foundational data everyone depends on.

  10. Step 10: Cross-charge on that basis. For usage-driven, weight the shared pipeline's cost by each consumer's downstream read/compute share:
-- consuming team's share of a shared pipeline, by downstream compute
      SELECT downstream_team,
      sum(compute) / sum(sum(compute)) OVER () AS charge_share
      FROM finops.lineage_consumption -- shared table -> consuming team -> compute
      WHERE shared_pipeline_id = :pipe
      GROUP BY 1;

11. Step 11: Leave genuinely common ingestion in a named shared-services cost centre that leadership owns — and show it as its own line. Never dissolve it silently into other teams; that's how the total stops reconciling and trust goes.

12. Step 12: Publish the cross-charge with the rule attached. Each consuming team sees not just what it was charged for shared data, but why and on what basis. The rule being visible is what makes people accept it.

It's not easy, it takes time. But doing it fully and properly makes IT department properly in charge and in control

Unable to connect to Tableau Cloud

dtu — Wed, 30 Sep 2026 12:54:39 GMT

I was trying to connect tables in Azure Databricks to Tableau Cloud using Explore In Tableau Cloud, but got the error:

Here's the instruction I follow Connect Tableau and Azure Databricks

I am the Tableau Cloud site admin, but don't see Explore in Tableau in Tableau Cloud settings. Did I miss anything? Appreciate any help.

Thanks.

Re: Dashboard DAB Deployment: default_catalog and default_schema do not work for metric views anymor

Capri — Wed, 30 Sep 2026 11:48:59 GMT

Thank you very much for the reply!

Fix 1 would work, but hard coding the catalog and schema name would not be a good solution for us, as we need different catalog names for dev and prod target.

Fix 2 would be great, but I cannot make it work, yet. I applied the changes as recommended and run `databricks bundle deploy`, but get the following error:

Error: invalid dependency "${var.catalog_name}", no such node ""

Re: Dashboard DAB Deployment: default_catalog and default_schema do not work for metric views anymor

Satyasai — Wed, 30 Sep 2026 10:39:42 GMT

This error occurs due to a recent update in strict validation enforcement in the Databricks Lakeview/Dashboard backend REST API (/api/2.0/lakeview/dashboards).

In the past, when passing dataset_catalog and dataset_schema via the query parameters, the backend allowed datasets inside .lvdash.json to specify a simple identifier (e.g., "test_metric_view") for asset_name and implicitly prepend the default catalog and schema.

Databricks has tightened the backend validation such that asset_name must always be a fully qualified 3-level Unity Catalog identifier (..), independent of whether dataset_catalog or dataset_schema are passed in the API call or DAB resource definition.

Recommended Fixes

Fix 1: Explicitly Fully-Qualify asset_name in the JSON File

Update the asset_name field in your dashboard.lvdash.json to explicitly use the full 3-level name:

JSON

{

"datasets": [

{

"name": "f9427c6f",

"displayName": "test_metric_view",

"asset_name": "<catalog_name>.<schema_name>.test_metric_view"

}

]

}

Fix 2: Parameterize catalog/schema using DAB Variables

If your bundle is deploying across multiple environments (e.g., dev, staging, prod) and uses dataset_catalog/dataset_schema to dynamically select environments, inject DAB variables into your JSON definition or convert it to a .json.tmpl file.

Option A: Using Databricks Asset Bundle Template variables (.tmpl)

Rename dashboard.lvdash.json to dashboard.lvdash.json.tmpl and reference your bundle variables directly:

JSON

{

"datasets": [

{

"name": "f9427c6f",

"displayName": "test_metric_view",

"asset_name": "${var.catalog_name}.${var.schema_name}.test_metric_view"

}

]

}

In databricks.yml:

YAML

resources:

dashboards:

test_dashboard:

display_name: "Bug Repro - Metric View Short Name"

file_path: ../dashboard.lvdash.json.tmpl

warehouse_id: ${var.warehouse_id}

targets:

dev:

variables:

catalog_name: dev_catalog

schema_name: dev_schema

prod:

variables:

catalog_name: prod_catalog

schema_name: prod_schema

Re: Recommended local development workflow for dashboard CI/CD with environment-specific catalog/sch

SamuelHarris — Wed, 30 Sep 2026 08:37:56 GMT

I too am facing the same issue, but my pipeline fails during the release. Previously the unqualified name was working and I moved my changes to Prod and its working fine there. I did move another lvdash dash file and now Im getting this error during release in devops:

validation failed: [[dashboard.datasets[d7278fb6].asset_name] invalid asset name.....(400 INVALID_PARAMETER_VALUE)

So, I tried this "asset_name": "${var.catalog}.my_schema.my_metric_view" then got this

Error: invalid dependency "${var.catalog}", no such node ""

From a reply above I understand that I need to give the unqualified name. But how do I solve the Invalid asset name error.

I follow all the same process mentioned in playnicekids's question.

Somebody help me sort this out.

Dashboard DAB Deployment: default_catalog and default_schema do not work for metric views anymore

Capri — Wed, 30 Sep 2026 08:13:49 GMT

We deploy Dashboards using DAB. We set default_catalog and default_schema for the dashboard, so in the `dashboard.lvdash.json` we only need the last part of table and metric names. Starting yesterday (Sep 29th, 2026), this does not work anymore and we get the following API error:

Error: cannot create resources.dashboards.test_dashboard: validation failed: [[dashboard.datasets[f9427c6f].asset_name] invalid asset name [asset_name = test_metric_view]] (400 INVALID_PARAMETER_VALUE) Endpoint: POST https://adb-<workspace_id>.azuredatabricks.net/api/2.0/lakeview/dashboards?dataset_catalog=<catalog_name>&dataset_schema=<schema_name> HTTP Status: 400 Bad Request API error_code: INVALID_PARAMETER_VALUE API message: validation failed: [[dashboard.datasets[f9427c6f].asset_name] invalid asset name [asset_name = test_metric_view]]

We have experienced this in at least 3 separate projects and cannot link the error to any changes on our end.

Minimal example to reproduce:
`dashboard.lvdash.json`:

{ "datasets": [ { "name": "f9427c6f", "displayName": "test_metric_view", # BUG: Short name instead of <catalog_name>.<schema_name>.test_metric_view "asset_name": "test_metric_view" } ], "pages": [ { "name": "c9748f1d", "displayName": "Test Page", "layout": [ { "widget": { "name": "3bdaafb0", "queries": [ { "name": "main_query", "query": { "datasetName": "f9427c6f", "fields": [ {"name": "id", "expression": "`id`"}, {"name": "name", "expression": "`name`"} ], "disaggregated": True } } ], "spec": { "version": 3, "widgetType": "bar", "encodings": { "x": {"fieldName": "id", "scale": {"type": "quantitative"}}, "y": {"fieldName": "name", "scale": {"type": "categorical"}} }, "data": {"queryName": "main_query"} } }, "position": {"x": 0, "y": 0, "width": 6, "height": 6} } ], "pageType": "PAGE_TYPE_CANVAS", "layoutVersion": "GRID_V1" } ] }

dashboard.yml:

resources: dashboards: test_dashboard: display_name: "Bug Repro - Metric View Short Name" file_path: ../dashboard.lvdash.json warehouse_id: ${var.warehouse_id} dataset_catalog: <catalog_name> dataset_schema: <schema_name>

The error occured within serverless environment version 5 and with the most recent Databricks CLI v1.18.0.

Re: Facing Data Truncation Issues in Databricks Dashboards

Prashant_Mishra — Thu, 24 Sep 2026 06:41:18 GMT

@NandiniN,

@Akshay_Petkar is probably talking about the limitation of pivot table data export rows. This is limited to 10k and not 1k. So, changing 1k setting will not help here. The underlying sql query returns 10k+ rows, but the pivot table UI truncates rows more than 10k. This is Databricks product feature. Is there a work around to get more then 10k records exported from pivot data?

Re: "Revoke" permissions for SQL-Warehouse with API

ashford — Tue, 22 Sep 2026 08:30:44 GMT

You may want to check whether the API expects a specific permission type for revocation rather than a special value such as NO_PERMISSIONS or REVOKE. The error suggests the request is reaching the endpoint, but the permission payload isn't matching the accepted schema.

I’d compare the revoke request against the API documentation/example for the exact SQL Warehouse permission type and payload format. Also check whether permissions are inherited from a catalog, schema, or group, since those may not be removable directly from the warehouse.

Re: ODBC v2.12- 503 "no Retry-After header"

saf_dbx — Mon, 21 Sep 2026 09:09:20 GMT

Thanks I will try setting the parameter and see if that will help. The databricks SQL Warehouse is already serverless in my case.

Re: ODBC v2.12- 503 "no Retry-After header"

ThomazNeto — Sun, 20 Sep 2026 11:03:08 GMT

Hi,

I've hit this exact error in a few environments, so maybe this helps.

First, the 503 itself is coming from the Databricks side, not from your BI tool or the driver. In every case I've traced it, the timestamps lined up with the SQL Warehouse being in a transitional state: auto-stop / cold start, scaling up or down, or a node being replaced. The service answers 503 for a few seconds and then recovers on its own. Worth cross-checking your failure times against the warehouse events and Query History.

What makes it fatal is the driver behavior. Out of the box, the ODBC driver only retries a 503 when the server includes a Retry-After header, and Databricks doesn't always send one. Without the header the driver gives up immediately and you get the Hardy (124) error.

About the parameter: EnableRetryWithoutRetryAfterHeader does exist, but you won't find it in the Databricks docs. I couldn't find it in the docs.databricks.com ODBC pages or in the release notes from 2.9.1 up to 2.12.1. The only place I've seen it written down is a Qlik support article from May this year, which says it's available from driver 2.9.1 onward and defaults to 0. It's one of those "key name only" options that don't show up in the DSN dialog.

To set it:

Windows: open regedit, navigate to HKEY_LOCAL_MACHINE, then SOFTWARE, then ODBC, then ODBC.INI, then your DSN name, and add a string value named EnableRetryWithoutRetryAfterHeader with value 1. Or append it to the connection string if your BI tool lets you.

Linux/macOS: add the line EnableRetryWithoutRetryAfterHeader=1 to the DSN section in odbc.ini.

DSN-less connection string: add EnableRetryWithoutRetryAfterHeader=1 as one more semicolon-separated property.

You're on 2.12.0, so version isn't a problem.

On the warehouse side, if this happens a lot, look at the auto-stop setting (extend it or disable it during the refresh window), or move to Serverless, which starts in seconds and cuts down on the cold-start 503s considerably. If you want proof for a support ticket, turn on driver logging (LogLevel=4 plus LogPath) and you should see the raw 503 with no Retry-After in the trace.

One caveat: since the parameter isn't officially documented by Databricks, I'd test it in your environment before rolling it out broadly. It's been stable where I've used it, but I can't point you to an official Databricks reference for it.

Hope this helps.

ODBC v2.12- 503 "no Retry-After header"

saf_dbx — Sun, 20 Sep 2026 08:08:26 GMT

I'm connecting to a Databricks catalog via a SQL Warehouse using the Databricks ODBC Driver (v2.12.0). During one of the fetches from my BI application, the transaction aborts with:

Database operation [SQLExecute] failed with error [[Databricks][Hardy] (124) A 503 response was returned but no Retry-After header was provided. Original error: Unknown]

I've seen references to a driver parameter called `EnableRetryWithoutRetryAfterHeader` that's supposed to make the driver retry a 503 even without a `Retry-After` header, but I can't find this documented in official Databricks ODBC docs.

This error doesn't come all the time, and most of the time it works okay.

Can anyone confirm:
If such a parameter exists, how can one configure this on the DSN settings?

Thanks in advance for any pointers.

Re: UC Delta API: managed-table READ_WRITE credentials rejected despite explicit User-Agent

GabFernandes — Tue, 15 Sep 2026 14:15:13 GMT

Hi @ablanchard .

The behavior you're seeing is expected for the Delta protocol-level credential endpoint (/api/2.1/unity-catalog/delta/v1/.../credentials). A few things to note:

Managed Delta table writes from external Delta clients are in Public Preview (docs)During this preview, the Delta protocol API validates the User-Agent header for READ_WRITE operations to gate access to recognized/registered connectors. This is why READ succeeds (it's GA) while READ_WRITE fails.
Consider using the general credential vending API instead. The documented public endpoint is POST /api/2.1/unity-catalog/temporary-table-credentials with a JSON body containing table_id(the table's UUID) and operation: "READ_WRITE". This is the officially documented path for external engines and may not enforce the same User-Agent gating. You can get the table UUID via the ListTables API with include_manifest_capabilities enabled — look for tables marked HAS_DIRECT_EXTERNAL_ENGINE_WRITE_SUPPORT.
Prerequisites: the metastore must have external_access_enabled = true, and the principal needs EXTERNAL_USE_SCHEMA on the parent schema. For catalog commits specifically, the table property delta.feature.catalogManaged must be set to 'supported'.
For the Delta protocol path specifically, connector registration/onboarding with Databricks is likely required during the Public Preview phase. Filing a feature request via the Databricks Ideas portal or opening a GitHub issue on the unitycatalog repo would be the best way to request formal onboarding for HarborSQL.
If my answer was helpful, please consider marking it as accepted solution!

Re: External embedding for reports using federated credentials fails

iamgoce — Mon, 14 Sep 2026 19:43:45 GMT

Databricks still hasn't committed to addressing this issue.

The only workaround is to get enrolled in the private preview for Dashboard Embedding with SSO (Embed with SSO Preview User Guide - Google Docs). This only works if you're already authorizing users with the same IdP SSO (e.g., Entra ID) and the users have the adequate access to the workspace, dashboard and SQL warehouse.

But for embedding dashboards for public consumption (non-Databricks users), the above won't work. One has to use a SP with a client secret.

UC Delta API: managed-table READ_WRITE credentials rejected despite explicit User-Agent

ablanchard — Sat, 12 Sep 2026 23:01:29 GMT

Hi everyone,

I’m developing HarborSQL, an open-source SQL engine using Apache DataFusion and delta-rs. I’m working on managed Delta table writes through the UC Delta API with catalog commits.

Using the same workspace, managed table, and user identity:

- Table metadata GET succeeds (200).
- Credentials GET with operation=READ succeeds (200).
- Credentials GET with operation=READ_WRITE fails (400).

This happens with both PAT and browser-based OAuth U2M authentication.

The failing request is:

GET /api/2.1/unity-catalog/delta/v1/catalogs/{catalog}/schemas/{schema}/tables/{table}/credentials?operation=READ_WRITE
User-Agent: HarborSQL_HarborSQL/0.1.9
Accept: application/json
Content-Type: application/json

The response says:

The provided User-Agent 'HarborSQL_HarborSQL/0.1.9' is insufficient.

It also directs connector developers to contact Databricks support, but I don’t have access to support.

The header follows the documented partner User-Agent format. Using harborsql/0.1.9 produces the same error. The failure occurs during credential vending, before any data write or commit request.

Does managed-table write access require connector registration, allowlisting, or additional preview enablement? Are there client-identification requirements beyond the documented User-Agent format?

If onboarding is required for independent open-source engines implementing catalog commits, could someone point me to the appropriate process or team?

Thanks!

Re: Controlling allow/ask/deny lists for predelivered MCP connectors

DB-RKL — Fri, 11 Sep 2026 14:21:14 GMT

Hey @Leembo

system.ai.microsoft_365 is now a 1st class UC object ("MCP service") that runs through Unity Gateway, same as the external connectors you register yourself.

Two levels of control:

- Whole connector on/off: an EXECUTE grant, so you can allow/deny M365 per user/group centrally today (GA).
- Denying specific tools tenant-wide w/ service policies: guardrails you attach to the MCP service to allow/deny/require-approval on individual tool calls. Service Policies are what enforce a per-tool denylist centrally so the per-user toggles can only pick within what you've allowed up top. (Beta)

I hope this helps

Controlling allow/ask/deny lists for predelivered MCP connectors

Leembo — Fri, 11 Sep 2026 10:03:27 GMT

Hello,

We are looking at utilizing the pre-delivered MCP connector to M365 via Genie One. The connector works great, but each user can individually configure the allowable tool calls without any central governance layer.

Are we missing something or is that currently a gap in pre-delivered connectors? Ideally, as an account administrator I would be able to select which tools are completely denied at a tenant level for users as a guardrail.

I know you can do that for externally registered MCP connectors by managing it at a catalog level, is that going to be possible for predelivered MCP connectors as well?

Thanks,
Linas M.

Re: Optimizing B2B E-Commerce & Construction Supply Chain Pipelines (Data Architecture Case Stud

SumeshKashyap — Thu, 10 Sep 2026 16:59:03 GMT

This one's really a network/download-client issue more than a Databricks one, but since the end goal is presumably getting the data into Databricks, a couple of angles:

- Browser downloads are single-connection and easy to bottleneck -- 8-9 Mbps on a 600 Mbps line is a classic sign of a single TCP stream being capped (by the server, a proxy, or per-connection throttling), not your bandwidth. A resumable, multi-connection downloader will usually do much better: "aria2c -x16 -s16 -c <url>" (parallel connections + auto-resume), or "curl -C - -O <url>" in a retry loop for simple resume without parallelism.

- Skip the local download entirely if you can. If Equinor's Volve data is hosted on a cloud object store (worth checking their data-sharing page for an S3/Azure Blob endpoint rather than an HTTPS file server), you can often ingest it straight into Databricks via Auto Loader or dbutils.fs.cp pointed at the bucket directly -- cloud-to-cloud transfer avoids your local network entirely and sidesteps this whole problem. Worth checking the dataset's distribution page for that option before spending days downloading 3+ TB to a laptop.

If it does have to come through your local connection, aria2c with resume + parallel connections is the standard fix for the 2-hour-timeout pattern you're describing.

Re: Optimizing B2B E-Commerce & Construction Supply Chain Pipelines (Data Architecture Case Stud

SumeshKashyap — Thu, 10 Sep 2026 16:58:03 GMT

@Gharhub Good questions!!!!

A few patterns that tend to work well for this kind of catalog:

Schema for heterogeneous raw materials: rather than one wide table per material category (which breaks every time a new attribute shows up), model a core materials table (id, category, supplier, base UoM) plus a category-specific attributes column stored as VARIANT (or MAP<STRING,STRING> if you're on an older Delta version) instead of literal EAV rows. That gets you schema flexibility without a proliferation of sparse columns, while keeping core dimensions (price, supplier, category) as real typed columns for fast filtering/joins.

Real-time-ish pricing: "real-time" freight/bulk pricing is rarely true streaming end-to-end -- usually it's: ingest price/freight-rate change events via Auto Loader into a Structured Streaming pipeline, apply pricing rules as a streaming MERGE into a current_prices Delta table (keyed by SKU+region+tier), and serve reads from that table directly if consumers tolerate seconds-old data, or via an online table/Lakebase if you need sub-100ms lookups from a checkout path. Keep the pricing rules as data in a rules table joined in, not hardcoded logic, since freight/bulk tiers change often.

DLT pipeline shape: standard medallion -- Bronze via Auto Loader off raw ERP/EDI drops, Silver applying the schema above plus dedup/CDC handling (APPLY CHANGES INTO if your source gives you CDC), Gold as the pricing-and-inventory marts your BI/pricing engine reads from. For inventory specifically, use APPLY CHANGES INTO with sequencing on your source's change timestamp so out-of-order updates from multiple warehouses resolve correctly instead of last-write-wins on arrival order.