Administration & Architecture topics

Does enabling Photon improve performance while lowering compute costs?

neerajdubey_86 — Fri, 04 Sep 2026 18:09:09 GMT

Please refer screen shot and help in clarifying if enabling Photon lead to lower execution costs?

Using Temporary Functions

Marbricks — Fri, 04 Sep 2026 17:31:51 GMT

Hi everyone,

I'm currently doing my professional internship and working on a reporting project that integrates with Databricks through Unity Catalog functions.

The reporting application is external to Databricks and uses Data Sources, Parameters, Result Sets, Grids, Charts, and Report Definitions to build dashboards. Existing reports consume data through functions referenced by a URI pattern similar to:

fn://catalog.schema.function_name@DATABRICKS.UNITYCATALOG
Show more lines

My mentor provided an existing report as a starting point and asked me to use it as a template for the new development. The report is already functional and contains the complete structure, including Data Sources, Parameters, Result Sets, Grids, Charts, and Report Definitions. My task is to understand how those pieces work together and then adapt or replace them to support a different dataset.

The challenge is that the new dataset does not currently have equivalent Unity Catalog functions available.

As guidance, my mentor suggested creating temporary functions first, using a pattern such as:

SQL
1
CREATE OR REPLACE TEMPORARY FUNCTION `catalog.schema.function_name`
2
RETURNS TABLE (...)
3
RETURN ...
4

Show more lines

The idea is to prototype the expected interface, validate the returned schema locally with the reporting application, and later create the permanent Unity Catalog functions if the design works correctly.

My mentor has already given me a clear direction, but as an intern I am still connecting the dots and trying to understand the complete flow from temporary functions, to production Unity Catalog functions, to the final report displayed in the application.

Has anyone worked on a similar integration between an external dashboard/reporting application and Databricks?

Any guidance, recommendations, or examples would be greatly appreciated.

I'm still pretty new to this area, so honestly I'm feeling a bit lost and trying to understand the correct path before I spend time going in the wrong direction 😅.

Thanks in advance! 👍

Marketplace Apps: Private Exchange Publishing, Enablement Requirements, and Review Process

bogdan_serban — Thu, 03 Sep 2026 11:32:13 GMT

Hello,

I'm looking for clarification regarding Databricks Marketplace Apps and Private Exchanges, as the currently available documentation appears to be inconsistent.

From the Marketplace API documentation, it appears that App is a supported Marketplace asset type (ASSET_TYPE_APP) https://docs.databricks.com/api/marketplaces/v1/get-listing, and Marketplace APIs also support concepts such as private listings, Private Exchanges, and listing visibility (PUBLIC/PRIVATE). Additionally, Marketplace APIs appear to treat Apps as first-class Marketplace assets alongside other listing types.

However, the provider-facing documentation for creating Marketplace listings currently documents asset types such as tables, volumes/files, models, notebooks, and MCP servers, but I have not found corresponding guidance for publishing an App through the Marketplace provider workflow. Likewise, the Private Exchange documentation generally discusses private listings and data products, but does not explicitly state whether App listings are supported within the Private Exchange model.

Could you please clarify the following:

Can a third-party provider publish an App asset type as a private listing within a Databricks Marketplace Private Exchange?
If App private listings are supported, is this capability available through the standard self-service Marketplace provider experience, or does it require account-level enablement, allowlisting, feature flags, or participation in a specific program?
If enablement is required, what is the correct process to request access to App listing capabilities?

The reason for the question is that the Marketplace APIs appear to expose App-related marketplace functionality, while the public provider documentation does not clearly describe whether Apps are included in the documented Private Exchange publishing flow. I'd appreciate confirmation of the supported model and links to any relevant documentation.

Thank you.

Who pays cross-cloud egress for Delta Sharing from GCP to Azure?

mariajsm — Tue, 01 Sep 2026 14:47:04 GMT

Hi DBX community!! Are cross-cloud egress charges for a Databricks-to-Databricks #DeltaShare billed to the provider account, and do they apply when the shared objects are views backed by Cloudflare?

I'm setting up a Delta Share using Databricks on GCP (us-central1) as the provider. The shared objects are views in Unity Catalog, and the underlying data is stored in Cloudflare R2.

The recipient is a customer using Databricks on Azure (West region). I'd like to understand how the costs are handled in this scenario:

Is there a cross-cloud egress/network charge when the Azure recipient queries the shared views?
If so, is that egress charge billed to the provider (GCP) or the recipient (Azure)?
Since the underlying data is stored in Cloudflare R2, does R2's lack of egress fees change or eliminate any of the Databricks cross-cloud egress charges?
Are there additional Databricks compute charges on the provider side when the recipient queries views through Delta Sharing?
Is there a recommended architecture to minimize costs if the recipient will query the shared data frequently?

Any clarification or real-world examples of how these charges appear in billing would be greatly appreciated.

Private listing of App Asset in Databricks Marketplace by 3rd Party Provider

AndreiaC — Tue, 01 Sep 2026 10:18:01 GMT

Hello,

Could you help confirm whether a third-party provider is able to publish an App asset type as a private listing within a Private Exchange?

If this is supported, is it a self-service capability, or does it need to be enabled per account? And if enablement is required, what's the correct way to request the App listing capability? What is the review process for private listing?

For context: the official documentation I've found on private listings so far focuses on data products (tables, volumes, models, notebooks, mcp). I haven't been able to locate provider-side guidance confirming that the App asset type is included in the private-listing / Private Exchange flow, so I'd appreciate clarification on whether Apps are covered — and a pointer to the relevant docs if they exist.

Thank you!

How to get my Database account verified?

Abarghodrat — Fri, 28 Aug 2026 23:19:38 GMT

Hi,

I am currently on a paid plan (Premium), but I can't find how to get my account verified. I need help and support team is referring me here to this forum to get help.

Very interesting, how support team can't help, not sure.

Anybody here who can help get my account verified?

What I need to resolve: At the moment, I can't have more than 3 Apps in my workspaces. Also my current Apps get stopped randomly as my account is not verified yet.

Thanks

Rez

Unable to delete schema with no tables

sapatel10 — Wed, 26 Aug 2026 19:30:06 GMT

I create Azure Databricks workspaces using Terraform. As a part of workspace creation I create the catalog and a schema in the catalog but not the tables. After this I follow below steps:

1. Create managed tables in the schema using Databricks workspace UI

2. Delete all managed tables in the schema using Databricks worksapce UI

3. Now deleting the schema using terraform or using Databricks UI is giving error that schema has managed table(s) and hence schema cannot be deleted. I need to know why?

Account API and Workspace API RBAC

Abishrp — Tue, 25 Aug 2026 13:29:19 GMT

Hi,

Currently, for my use case, I am using a service principal with Account Admin permissions to call account-level APIs. For workspace-level APIs, I am using a separate service principal with Workspace Admin permissions.

Is there a way to avoid assigning the full Account Admin or Workspace Admin role to these service principals and instead grant only the minimum permissions required for the specific APIs they need to call?

I would also like to ensure that each service principal has access only to the required APIs and operations. For example, the account-level service principal should have only the necessary read/write permissions for the specific account-level APIs it uses, rather than having full Account Admin access.

Similarly, the workspace-level service principal should have only the required permissions for the specific workspace-level APIs.

I am looking for a least-privilege approach for assigning permissions to these service principals.

How do ISVs prevent customers modifying a wheel deployed into their own Databricks workspace?

vamsi_simbus — Tue, 25 Aug 2026 11:49:30 GMT

We're an ISV. Our product is a DLT pipelines, batch jobs, a Databricks App, and MLflow
models — is installed into each customer's own Databricks workspace as a dedicated
deployment, using Asset Bundles with one target per customer. We maintain the code
centrally; customers never receive the repo. We're packaging it as a wheel and running
it via python_wheel_task.

What we need is integrity, not secrecy. We accept that a workspace admin can read the
code. What matters is that the code we shipped is the code that runs — so that when
results are wrong it's our bug, and when we report outcomes, they're attributable to our
logic rather than to a local change.

How are others handling this when the deployment lives in a workspace the customer
administers? Interested in what has worked in practice, and where you decided the extra
effort stopped being worth it.

What should I do if a business payment is made by cheque and the cheque gets bounced?

advocatesiddhar — Sat, 22 Aug 2026 06:53:34 GMT

I’m looking for some general information about handling a situation where a business payment was made through a cheque, but the cheque was later returned unpaid by the bank.

What are the usual steps after a cheque is dishonoured? For example, when should the bank return memo, payment records, and other documents be collected, and when is a legal notice generally considered appropriate?

I’m particularly interested in understanding how cheque bounce cases are normally handled in India and what someone should consider before starting legal proceedings. It would also be useful to know whether consulting a lawyer early can help avoid procedural mistakes.

I came across Nair Law Chamber, which discusses cheque bounce matters and the legal process involved, and I’m interested in hearing how others have approached similar situations.

Estimating Databricks Annual Implementation and Licensing Costs in 2026

nafikazi — Thu, 20 Aug 2026 02:02:00 GMT

Hello everyone,

Good evening. I am a data analyst and a Databricks Certified Data Engineer Associate. While I understand the platform from a technical perspective and recognize its value, I am still learning how to estimate the full financial impact of a Databricks implementation.

When discussing modernization with business leaders, questions often arise about migration, implementation, platform consumption, cloud infrastructure, support, training, and ongoing operational costs. I do not yet have a complete understanding of how these costs are calculated and presented.

To improve my understanding, I prepared a one-page document outlining the potential costs of migrating existing Power BI Dataflow Gen1, Tableau Prep, and Alteryx ETL workflows to the Databricks ecosystem.

If you have time, I would sincerely appreciate your review and feedback. Please let me know if I have missed any major cost components, assumptions, or potential expenses.

My main question is: For a medium-sized business, how can we reasonably estimate the total cash outflow required to implement, migrate to, and operate Databricks?

Any practical costing framework, real-world example, or guidance would be greatly appreciated. My goal is to strengthen both my technical and commercial understanding as I work toward becoming a Databricks consultant, prepare for future interviews, and evaluate modernization projects more confidently.

Thanks
Nafi

Microsoft Fabric One and Databricks Unity Catalog — Bidirectional Access Without Data Movement

amitsharma1707 — Wed, 19 Aug 2026 13:09:47 GMT

Hi everyone,

We're working in a hybrid environment with both Microsoft Fabric and Databricks, and I want to understand the options for virtualizing data between the two platforms without physically moving or duplicating it — similar to how shortcuts work in Fabric OneLake.

Specifically, I'm looking for guidance on both directions:

1. Accessing Fabric OneLake data from Databricks

What's the recommended way to read Delta tables stored in OneLake directly from Databricks (e.g., registering OneLake paths as external tables in Unity Catalog via ABFS, using external locations/storage credentials)?
Are there limitations with Unity Catalog external locations pointing to OneLake (authentication, service principals, workspace-level access)?
Can OneLake data be governed under Unity Catalog while the data physically stays in OneLake?

2. Accessing Databricks Unity Catalog data from Fabric (like a shortcut)

What's the current experience with OneLake shortcuts to ADLS Gen2 where Databricks Delta tables reside?
Has anyone used the Fabric mirroring for Azure Databricks Unity Catalog feature? How well does it reflect UC metadata and keep tables in sync without copying data?
Any gotchas with Delta table features (deletion vectors, column mapping, liquid clustering) breaking readability on the Fabric side?

3. General considerations

Performance implications of cross-platform virtualization vs. native storage
How governance/permissions behave when the same data is visible in both Unity Catalog and OneLake
Real-world experiences: does this dual-visibility setup work reliably in production?
@Genpact

Account Console environment for demonstrating workspace management capabilities to clients ?

rkhand14_ltm — Tue, 18 Aug 2026 06:12:31 GMT

Hi Databricks Community,

Is there a way for Databricks partners to obtain access to an Account Console environment for demonstrating workspace management capabilities to clients?

We are looking for a sandbox, free-tier environment, or any partner lab setup that would allow us to:

Build internal capability and hands-on expertise
Develop and demonstrate proof-of-concepts (POCs)
Showcase workspace administration and governance features to prospective clients
Support Data Warehouse Modernization discussions and help convert leads into opportunities

Any guidance, partner programs, or available demo environments would be greatly appreciated.

Thank you!

Azure Databricks Serverless Compute Unable to Connect to Azure SQL MI Using Failover Group FQDN via

LokeshChikuru — Mon, 17 Aug 2026 14:16:58 GMT

We are using Azure Databricks Serverless Compute to connect to Azure SQL Managed Instance (SQL MI) through Network Connectivity Configuration (NCC) with Private Endpoint Rules. Connectivity works successfully when using the primary SQL MI FQDN. However, connectivity fails when the SQL Managed Instance is configured behind a Failover Group and the connection is attempted using the Failover Group listener FQDN. As a result, Databricks Serverless Compute is currently unable to establish a connection to Azure SQL MI via the Failover Group FQDN, while direct connections to the primary SQL MI endpoint continue to work as expected.

Billing

kosarchik — Fri, 14 Aug 2026 13:54:28 GMT

Hello, where can I find a breakdown of usage from the monitoring.azure_costs.costs_details table? I filtered MeterCategory = 'Foundry Models' to find out who is using which models, but the data there is only aggregated.

NetSuite JDBC Driver 8.10.190.0 - Databricks support

xx1datageek — Thu, 13 Aug 2026 01:13:11 GMT

Hi,

Currently I'm trying to connect my databricks to Netsuite using Lakeflow connect. At the moment, Netsuite only provides JAR driver version 8.10.190.0. Would Databricks be supporting this version soon? If not yet, can I know when would Databricks be able to do so ? The reason of why I'm not trying to use the older versions is because in this version Netsuite has done some fix on some issues that occured in the older versions.

Could you please shed some light on this ??

Cheers!

X_NHC_CONTROL_PLANE_UNREACHABLE

taglud_dbw — Wed, 12 Aug 2026 20:51:18 GMT

I have this issue

[details] X_NHC_CONTROL_PLANE_UNREACHABLE: Instance failed network health check before bootstrapping with fatal error: X_NHC_CONTROL_PLANE_UNREACHABLE 2 failed component(s): control_plane internet Retryable: true Based on the failure results: List(entity: "**region**-c2.azuredatabricks.net" outcome: "unreachable" duration_sec: 281.2346 message: "curl: (28) Connection timed out after 10000 milliseconds" last_error_code: 28 , entity: "www.databricks.com" outcome: "unreachable" duration_sec: 225.03764 message: "curl: (28) Failed to connect to www.databricks.com port 443 after 7603 ms: Connection timed out" last_error_code: 28 )(OnDemand)

i did all the network check recommanded by AI

No custom route tables

No custom DNS

No failed Azure deployments

Correct NSG rules
what could be the issue in your opinion ? should i recreate the resource ?

Unity Catalog Metric Views to be accessible to Custom Apps outside of DBX environment

vkondepati — Wed, 12 Aug 2026 15:12:41 GMT

Hello Everyone,

We have a requirement to expose Databricks Unity Catalog Metric Views to custom applications running outside the Databricks environment.

The application team would like to consume governed KPIs and metrics through APIs such as REST or GraphQL, rather than accessing them directly through Databricks applications or SQL interfaces.

I’d like to understand the recommended Databricks architecture for this use case:

Does Databricks provide an out-of-the-box REST or GraphQL API for querying Unity Catalog Metric Views from external applications?
If not, is the recommended approach to build a custom API/service layer that queries the Metric Views through Databricks SQL?
For applications requiring high concurrency and millisecond-level response times, would it make sense to materialize or synchronize the calculated metric data into Lakebase and expose it through an application/API layer?
Would this introduce duplication between the governed Metric Views in Unity Catalog and the serving layer in Lakebase? If so, what is the recommended approach for keeping the metric definitions and serving data consistent?
From an architecture perspective, which pattern would Databricks recommend:
Custom App → API → Databricks SQL → UC Metric Views
or
Custom App → API → Lakebase ← Metric Views / Databricks pipelines?
Are there other Databricks-native capabilities better suited for providing low-latency, highly concurrent access to governed metrics for external applications?

The main objective is to maintain Unity Catalog Metric Views as the governed semantic/metrics layer, while providing external applications with a scalable, low-latency API-based consumption model.

Regards,
Venkat

Databricks App architecture with AppKit - agentic app with governed write-back functionality

Fortinbras — Wed, 12 Aug 2026 10:12:02 GMT

Hi Everyone,

I am currently working on an internal application for editing and analyzing forecast data built and hosted via Databricks Apps. I would appreciate a sanity check from anyone who has taken AppKit into production as I am planning on utilizing the official AppKit TypeScript SDK to leverage pre-built plugins, alongside agent orchestration using LangChain Python SDK behind a Model Serving endpoint.

The users would ask an LLM-driven conversational agent questions about forecast data, based on which the agent would be able to query the relevant data and get back tables and charts. The trickier part is that application should allow users to edit and modify the returned data either by natural-language prompts or directly in an interactive table, and then those overrides should be written back to the underlying tables.

Are there any established AppKit patterns or plugins for transactional write-back and optimistic concurrency control and audit history, or more bespoke solutions are needed. My current thinking is to use Lakebase via its plugin and an append-only history table. Perhaps a Python-based architecture (e.g. Dash) with hand-build flows would be more suitable?

Thanks in advance for any architecture examples, lessons learned, or documentation pointers.

Azure Databricks Default Package Repository with Azure Key Vault-backed Secret Scope

kfadratek — Tue, 11 Aug 2026 08:19:09 GMT

Hi,

I’m trying to configure Default Python Package Repository in an Azure Databricks workspace and would like to clarify whether the predefined secret scope can be backed by Azure Key Vault.

According to the documentation, Default Package Repository can use the predefined secret scope:

databricks-package-management

with the following keys:

pip-index-url
pip-extra-index-urls
pip-cert

The documentation shows an example using:

databricks secrets create-scope databricks-package-management followed by put-secret commands.

However, I cannot find an explicit statement saying that databricks-package-management must be a Databricks-backed secret scope.

The documentation also describes two types of secret scopes, including Azure Key Vault-backed scopes. An Azure Key Vault-backed scope is read-only from the Databricks side, so the secrets would be created directly in Azure Key Vault rather than using databricks secrets put-secret.

My question is:

Has anyone successfully configured Default Package Repository using databricks-package-management as an Azure Key Vault-backed secret scope?

I have already tested this configuration. The Key Vault-backed scope exists and contains pip-index-url, and Databricks can read the secret through the scope.

However, I cannot specify the secret scope in the Default Package Repositories UI, and I would like to understand whether Databricks automatically resolves the predefined databricks-package-management scope or whether this functionality specifically requires a Databricks-backed scope.

Has anyone tested this configuration and got it working?

Alternatively, does anyone have official information or a technical reference confirming that databricks-package-management must use a Databricks-backed scope and cannot use an Azure Key Vault-backed scope?

Thanks!