cancel
Showing results for 
Search instead for 
Did you mean: 
Administration & Architecture
Explore discussions on Databricks administration, deployment strategies, and architectural best practices. Connect with administrators and architects to optimize your Databricks environment for performance, scalability, and security.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Kaz1
by New Contributor
  • 2487 Views
  • 1 replies
  • 0 kudos

Can't post to microsoft teams workflow from databricks notebook

When trying to post to a microsoft teams webhook, from a databricks notebook, using compute with DBR 12.2 I receive the following error:SSL error: HTTPSConnectionPool(host='prod-104.westeurope.logic.azure.com', port=443): Max retries exceeded with ur...

  • 2487 Views
  • 1 replies
  • 0 kudos
Latest Reply
Isi
Honored Contributor III
  • 0 kudos

Hello @Kaz1 It’s very likely that the issue is related to where the HTTPS request originates — whether it’s coming from the Databricks control plane or your data plane (your own AWS VPC).When you run a local script or call the Teams webhook from a cl...

  • 0 kudos
m2chrisp
by New Contributor II
  • 779 Views
  • 1 replies
  • 1 kudos

Resolved! Setting catalog isolation mode and workspace bindings within a notebook using Python SDK

Hi,I have a set of notebooks which configure new catalogs, set permissions, create default schemas, attach Azure Storage accounts as external volumes, create Git Folders and set current branches, etc.All this works just fine.One thing I'm trying to a...

  • 779 Views
  • 1 replies
  • 1 kudos
Latest Reply
mark_ott
Databricks Employee
  • 1 kudos

The error occurs because the Databricks Python SDK (databricks-sdk) and the authentication method within an Azure Databricks notebook use a special “db-internal” token for user-based notebook execution, which does not have permission to perform some ...

  • 1 kudos
IoanT
by New Contributor
  • 644 Views
  • 1 replies
  • 2 kudos

Resolved! databricks bundle validate: Recommendation: permissions section should explicitly include the curren

Starting from 10/07/2025 my validation bundle step from databricks bundle deploy fail with the folowing message:2025-07-11T07:07:18.5175554Z Recommendation: permissions section should explicitly include the current deployment identity '***' or one of...

  • 644 Views
  • 1 replies
  • 2 kudos
Latest Reply
mark_ott
Databricks Employee
  • 2 kudos

The error message in your Databricks bundle deploy validation step:   text Recommendation: permissions section should explicitly include the current deployment identity '***' or one of its groups If it is not included, CAN_MANAGE permissions are...

  • 2 kudos
Luke_Kociuba
by New Contributor
  • 434 Views
  • 3 replies
  • 5 kudos

dbt+Databrics

Hi!I will use dbt + Databrics in my new project.I'm runing dbt training. Have Databrics free/trial account so far. Want to connect, link dbt with Databbrics and create data warehouse from below data setsjaffle_shop_customers.csvjaffle_shop_orders.csv...

  • 434 Views
  • 3 replies
  • 5 kudos
Latest Reply
BS_THE_ANALYST
Esteemed Contributor II
  • 5 kudos

Great solution @szymon_dybczak .I don't know too much about DBT yet. Is there much difference between connecting/interacting with DBT cloud vs DBT core?All the best,BS

  • 5 kudos
2 More Replies
HitMah
by New Contributor II
  • 442 Views
  • 3 replies
  • 2 kudos

Transitioning Approach for Evolving EDW

As EDW will continue to evolve with new data and business logic during the multi-phased migration, what architectural strategies and design patterns can minimize rework when migrating from an evolving Enterprise Data Warehouse (EDW) to Databricks? I ...

  • 442 Views
  • 3 replies
  • 2 kudos
Latest Reply
-werners-
Esteemed Contributor III
  • 2 kudos

There is no single approach. It depends on your organization.First you have the 'impact-axis' which is lean and mean vs big bang.Next you also have bottom-up (first fix bronze and work upwards) or top-down (focus on gold and read data from your legac...

  • 2 kudos
2 More Replies
danielrodri
by New Contributor III
  • 1626 Views
  • 6 replies
  • 1 kudos

Resolved! Payment receipts of Databricks payments

Hello experts,I am trying to get receipts for the monthly payments done to Databricks. I need them for the financial department of the organization I am working for. The only billing information I get access to is the usage dashboards and the tables ...

  • 1626 Views
  • 6 replies
  • 1 kudos
Latest Reply
DatabricksEddy
New Contributor II
  • 1 kudos

Hello everyone!I'd like to know how I can resolve a payment issue with Databricks?My credit card is registered to be charged automatically, but it hasn't been charged, as I received a message from Databricks asking me to make the payment, etc.Can any...

  • 1 kudos
5 More Replies
jzu
by New Contributor
  • 286 Views
  • 1 replies
  • 1 kudos

Resolved! Problem with Metastore

Hello community.We are facing an issue when deploying and configuring metastore using terraform. We are using Azure Devops pipeline for deployment. The identity running the pipeline is a managed identity and it's set as account admin in Account porta...

  • 286 Views
  • 1 replies
  • 1 kudos
Latest Reply
Louis_Frolio
Databricks Employee
  • 1 kudos

Greetings @jzu , I did some digging around with internal docs and references and put together some helpful tips and things to consider.   This is a common authorization issue related to permission propagation delays and ownership configuration when m...

  • 1 kudos
Awoke101
by New Contributor III
  • 3733 Views
  • 1 replies
  • 0 kudos

Resolved! Ray cannot detect GPU on the cluster

I am trying to run ray on databricks for chunking and embedding tasks. The cluster I’m using is:g4dn.xlarge1-4 workers with 4-16 cores1 GPU and 16GB memoryI have set spark.task.resource.gpu.amount to 0.5 currently.This is how I have setup my ray clus...

  • 3733 Views
  • 1 replies
  • 0 kudos
Latest Reply
Krishna_S
Databricks Employee
  • 0 kudos

I have replicated all your steps and created the ray cluster exactly as you have done. Also, I have set: spark.conf.set("spark.task.resource.gpu.amount", "0.5") And I see a warning that shows that I don't allocate any GPU for Spark (as 1), even tho...

  • 0 kudos
noorbasha534
by Valued Contributor II
  • 1003 Views
  • 4 replies
  • 2 kudos

Oauth Token federation

Dear allHas anyone tried oauth token federation for authentication with Databricks REST APIs?appreciate if there is a re-usable code snippet to achieve the same.

  • 1003 Views
  • 4 replies
  • 2 kudos
Latest Reply
nayan_wylde
Honored Contributor III
  • 2 kudos

@noorbasha534  Here is a sample python code I use for getting oauth token from Azure Active Directory and then pass the token in databricks API. Prerequisite is the SPN needs to be a admin in the workspace.import requests # Azure AD credentials tena...

  • 2 kudos
3 More Replies
ekmazars
by New Contributor II
  • 333 Views
  • 2 replies
  • 1 kudos

Resolved! SQLSTATE HY000 after upgrading from Databricks 15.4 to 16.4

After upgrading from Databricks 15.4 to 16.4, without changing our Python code, we suddenly get SQL Timeouts, see below.Is there some new timeout default, that we don't know about, that we need to increase with the new version? After a quick search I...

  • 333 Views
  • 2 replies
  • 1 kudos
Latest Reply
mark_ott
Databricks Employee
  • 1 kudos

After upgrading to Databricks 16.4, there is a notable change in SQL timeout behavior. The default timeout for SQL statements and objects like materialized views and streaming tables is now set to two days (172,800 seconds). This system-wide default ...

  • 1 kudos
1 More Replies
noorbasha534
by Valued Contributor II
  • 395 Views
  • 2 replies
  • 0 kudos

View Refresh Frequency

Dear allwe have around 5000+ finished data products (aka views) in several schemas of unity catalog. One question that comes from business users frequently is - how frequently these get refreshed?for that the answer is not simpler as the underlying t...

  • 395 Views
  • 2 replies
  • 0 kudos
Latest Reply
saurabh18cs
Honored Contributor II
  • 0 kudos

Hi @noorbasha534 just a pseudocode:for view in all_views:lineage = get_lineage(view) # Use Unity Catalog APIbase_tables = extract_base_tables(lineage)refresh_times = []for table in base_tables:job = find_job_refreshing_table(table) # Custom logic/met...

  • 0 kudos
1 More Replies
faaiqsaheed12
by New Contributor II
  • 12635 Views
  • 12 replies
  • 7 kudos

Resolved! Unable to view Manage Account Option in Databricks UI

Hi All, I want to remove Unity Catalog from Admin Console so that I need to navigate to manage account option but the option is not available in my Databricks workspace. Please help me to sort this issue by removing Unity Catalog.

  • 12635 Views
  • 12 replies
  • 7 kudos
Latest Reply
Amit_Dass
New Contributor II
  • 7 kudos

Brilliant! I followed the above and its works seamlessly. 

  • 7 kudos
11 More Replies
robert-moyai
by New Contributor II
  • 594 Views
  • 3 replies
  • 2 kudos

Databricks OAuth: User-based OAuth (U2M) Databricks Connect in Apps

I'm looking to use a databricks session in a Databricks app. The databricks session should be able to use user-based oauth (U2M) to ensure the app has same privileges as the authenticated user using the app. Databricks apps have the ability to use th...

  • 594 Views
  • 3 replies
  • 2 kudos
Latest Reply
robert-moyai
New Contributor II
  • 2 kudos

Thanks for you response and the links. But the documentation doesn't explicitly explain why the spark connect has been placed out of scope and what app builders should use to implement proper data governance using on behave of user permissions.

  • 2 kudos
2 More Replies
trailblazer
by New Contributor II
  • 1658 Views
  • 5 replies
  • 3 kudos

Resolved! Connecting Azure databricks with firewall enabled Azure storage account

Hi I am trying to connect from Azure Databrick workspace to Azure gen2 storage account securely. The storage account is set up with these options1. Enabled from selected virtual networks and IP addresses- we whitelisted few ips 2. Added Microsoft.Dat...

  • 1658 Views
  • 5 replies
  • 3 kudos
Latest Reply
mkkao924
New Contributor II
  • 3 kudos

I am having exact issue as @trailblazer , that if I enable traffic for all network, I can read/write to storage account, if I only allow selected network, including the VNet, then it doesn't. I am using Serverless setup. I also followed the firewall ...

  • 3 kudos
4 More Replies
juan_maedo
by New Contributor III
  • 397 Views
  • 1 replies
  • 1 kudos

Resolved! Job Notifications specifically on Succeeded with Failures

Hi everyone,I have a set of jobs that always execute the last task regardless of whether the previous ones failed or not (using the ‘ALL done’ execution dependency).When moving to production and wanting to enable notifications, there is no option to ...

  • 397 Views
  • 1 replies
  • 1 kudos
Latest Reply
mark_ott
Databricks Employee
  • 1 kudos

Databricks does not provide a direct way to distinguish or send notifications specifically for a "Succeeded with failures" state at the job level—the job is classified as "Success" even when some upstream tasks have failed, if the last (leaf) task is...

  • 1 kudos