cancel
Showing results for 
Search instead for 
Did you mean: 
Administration & Architecture
Explore discussions on Databricks administration, deployment strategies, and architectural best practices. Connect with administrators and architects to optimize your Databricks environment for performance, scalability, and security.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

HitMah
by New Contributor II
  • 415 Views
  • 3 replies
  • 2 kudos

Transitioning Approach for Evolving EDW

As EDW will continue to evolve with new data and business logic during the multi-phased migration, what architectural strategies and design patterns can minimize rework when migrating from an evolving Enterprise Data Warehouse (EDW) to Databricks? I ...

  • 415 Views
  • 3 replies
  • 2 kudos
Latest Reply
-werners-
Esteemed Contributor III
  • 2 kudos

There is no single approach. It depends on your organization.First you have the 'impact-axis' which is lean and mean vs big bang.Next you also have bottom-up (first fix bronze and work upwards) or top-down (focus on gold and read data from your legac...

  • 2 kudos
2 More Replies
danielrodri
by New Contributor III
  • 1617 Views
  • 6 replies
  • 1 kudos

Resolved! Payment receipts of Databricks payments

Hello experts,I am trying to get receipts for the monthly payments done to Databricks. I need them for the financial department of the organization I am working for. The only billing information I get access to is the usage dashboards and the tables ...

  • 1617 Views
  • 6 replies
  • 1 kudos
Latest Reply
DatabricksEddy
New Contributor II
  • 1 kudos

Hello everyone!I'd like to know how I can resolve a payment issue with Databricks?My credit card is registered to be charged automatically, but it hasn't been charged, as I received a message from Databricks asking me to make the payment, etc.Can any...

  • 1 kudos
5 More Replies
Angus-Dawson
by New Contributor III
  • 1165 Views
  • 4 replies
  • 3 kudos

Databricks Runtime 16.4 LTS has inconsistent Spark and Delta Lake versions

Per the release notes for Databricks Runtime 16.4 LTS, the environment has Apache Spark 3.5.2 and Delta Lake 3.3.1:https://docs.databricks.com/aws/en/release-notes/runtime/16.4ltsHowever, Delta Lake 3.3.1 is built on Spark 3.5.3; the newest version o...

  • 1165 Views
  • 4 replies
  • 3 kudos
Latest Reply
saurabh18cs
Honored Contributor II
  • 3 kudos

Hi @Angus-Dawson Use  Databricks Connect for local development/testing against a remote Databricks cluster—this ensures your code runs in the actual Databricks environment and databricks managed dbrs which are different from open-source versions((DBR...

  • 3 kudos
3 More Replies
jzu
by New Contributor
  • 279 Views
  • 1 replies
  • 1 kudos

Resolved! Problem with Metastore

Hello community.We are facing an issue when deploying and configuring metastore using terraform. We are using Azure Devops pipeline for deployment. The identity running the pipeline is a managed identity and it's set as account admin in Account porta...

  • 279 Views
  • 1 replies
  • 1 kudos
Latest Reply
Louis_Frolio
Databricks Employee
  • 1 kudos

Greetings @jzu , I did some digging around with internal docs and references and put together some helpful tips and things to consider.   This is a common authorization issue related to permission propagation delays and ownership configuration when m...

  • 1 kudos
Awoke101
by New Contributor III
  • 3701 Views
  • 1 replies
  • 0 kudos

Resolved! Ray cannot detect GPU on the cluster

I am trying to run ray on databricks for chunking and embedding tasks. The cluster I’m using is:g4dn.xlarge1-4 workers with 4-16 cores1 GPU and 16GB memoryI have set spark.task.resource.gpu.amount to 0.5 currently.This is how I have setup my ray clus...

  • 3701 Views
  • 1 replies
  • 0 kudos
Latest Reply
Krishna_S
Databricks Employee
  • 0 kudos

I have replicated all your steps and created the ray cluster exactly as you have done. Also, I have set: spark.conf.set("spark.task.resource.gpu.amount", "0.5") And I see a warning that shows that I don't allocate any GPU for Spark (as 1), even tho...

  • 0 kudos
noorbasha534
by Valued Contributor II
  • 997 Views
  • 4 replies
  • 2 kudos

Oauth Token federation

Dear allHas anyone tried oauth token federation for authentication with Databricks REST APIs?appreciate if there is a re-usable code snippet to achieve the same.

  • 997 Views
  • 4 replies
  • 2 kudos
Latest Reply
nayan_wylde
Honored Contributor III
  • 2 kudos

@noorbasha534  Here is a sample python code I use for getting oauth token from Azure Active Directory and then pass the token in databricks API. Prerequisite is the SPN needs to be a admin in the workspace.import requests # Azure AD credentials tena...

  • 2 kudos
3 More Replies
ekmazars
by New Contributor II
  • 330 Views
  • 2 replies
  • 1 kudos

Resolved! SQLSTATE HY000 after upgrading from Databricks 15.4 to 16.4

After upgrading from Databricks 15.4 to 16.4, without changing our Python code, we suddenly get SQL Timeouts, see below.Is there some new timeout default, that we don't know about, that we need to increase with the new version? After a quick search I...

  • 330 Views
  • 2 replies
  • 1 kudos
Latest Reply
mark_ott
Databricks Employee
  • 1 kudos

After upgrading to Databricks 16.4, there is a notable change in SQL timeout behavior. The default timeout for SQL statements and objects like materialized views and streaming tables is now set to two days (172,800 seconds). This system-wide default ...

  • 1 kudos
1 More Replies
noorbasha534
by Valued Contributor II
  • 389 Views
  • 2 replies
  • 0 kudos

View Refresh Frequency

Dear allwe have around 5000+ finished data products (aka views) in several schemas of unity catalog. One question that comes from business users frequently is - how frequently these get refreshed?for that the answer is not simpler as the underlying t...

  • 389 Views
  • 2 replies
  • 0 kudos
Latest Reply
saurabh18cs
Honored Contributor II
  • 0 kudos

Hi @noorbasha534 just a pseudocode:for view in all_views:lineage = get_lineage(view) # Use Unity Catalog APIbase_tables = extract_base_tables(lineage)refresh_times = []for table in base_tables:job = find_job_refreshing_table(table) # Custom logic/met...

  • 0 kudos
1 More Replies
faaiqsaheed12
by New Contributor II
  • 12617 Views
  • 12 replies
  • 7 kudos

Resolved! Unable to view Manage Account Option in Databricks UI

Hi All, I want to remove Unity Catalog from Admin Console so that I need to navigate to manage account option but the option is not available in my Databricks workspace. Please help me to sort this issue by removing Unity Catalog.

  • 12617 Views
  • 12 replies
  • 7 kudos
Latest Reply
Amit_Dass
New Contributor II
  • 7 kudos

Brilliant! I followed the above and its works seamlessly. 

  • 7 kudos
11 More Replies
robert-moyai
by New Contributor II
  • 575 Views
  • 3 replies
  • 2 kudos

Databricks OAuth: User-based OAuth (U2M) Databricks Connect in Apps

I'm looking to use a databricks session in a Databricks app. The databricks session should be able to use user-based oauth (U2M) to ensure the app has same privileges as the authenticated user using the app. Databricks apps have the ability to use th...

  • 575 Views
  • 3 replies
  • 2 kudos
Latest Reply
robert-moyai
New Contributor II
  • 2 kudos

Thanks for you response and the links. But the documentation doesn't explicitly explain why the spark connect has been placed out of scope and what app builders should use to implement proper data governance using on behave of user permissions.

  • 2 kudos
2 More Replies
trailblazer
by New Contributor II
  • 1636 Views
  • 5 replies
  • 3 kudos

Resolved! Connecting Azure databricks with firewall enabled Azure storage account

Hi I am trying to connect from Azure Databrick workspace to Azure gen2 storage account securely. The storage account is set up with these options1. Enabled from selected virtual networks and IP addresses- we whitelisted few ips 2. Added Microsoft.Dat...

  • 1636 Views
  • 5 replies
  • 3 kudos
Latest Reply
mkkao924
New Contributor II
  • 3 kudos

I am having exact issue as @trailblazer , that if I enable traffic for all network, I can read/write to storage account, if I only allow selected network, including the VNet, then it doesn't. I am using Serverless setup. I also followed the firewall ...

  • 3 kudos
4 More Replies
juan_maedo
by New Contributor III
  • 375 Views
  • 1 replies
  • 1 kudos

Resolved! Job Notifications specifically on Succeeded with Failures

Hi everyone,I have a set of jobs that always execute the last task regardless of whether the previous ones failed or not (using the ‘ALL done’ execution dependency).When moving to production and wanting to enable notifications, there is no option to ...

  • 375 Views
  • 1 replies
  • 1 kudos
Latest Reply
mark_ott
Databricks Employee
  • 1 kudos

Databricks does not provide a direct way to distinguish or send notifications specifically for a "Succeeded with failures" state at the job level—the job is classified as "Success" even when some upstream tasks have failed, if the last (leaf) task is...

  • 1 kudos
MiriamHundemer
by New Contributor III
  • 476 Views
  • 2 replies
  • 3 kudos

Resolved! Error when trying to destory databricks_permissions with OpenTofu

Hi,In our company's project we created a databricks_user for a service account (which is needed for our deployment process) via OpenTofu and afterwards adjusted permissions to that "user's" user folder using the databricks_permissions resource.resour...

  • 476 Views
  • 2 replies
  • 3 kudos
Latest Reply
NandiniN
Databricks Employee
  • 3 kudos

Hi @MiriamHundemer , The issue occurs because the owner of the home folder (in this case, the databricks_user.databricks_deployment_sa service account) often has an unremovable CAN_MANAGE permission on its own home directory. When OpenTofu attempts t...

  • 3 kudos
1 More Replies
Hil
by New Contributor II
  • 320 Views
  • 4 replies
  • 1 kudos

Resolved! Deply databricks workspace on azure with terraform - failed state: legacy access

I'm trying to deploy a workspace on azure via terraform and i'm getting the following error:"INVALID_PARAMETER_VALUE: Given value cannot be set for workspace~<id>~default_namespace_ws~ because: cannot set default namespace to hive_metastore since leg...

  • 320 Views
  • 4 replies
  • 1 kudos
Latest Reply
Hil
New Contributor II
  • 1 kudos

I found the issue, The setting automatically assigned workspaces to this metastore was checked. Unchecking this and manually assigning the metastore worked.

  • 1 kudos
3 More Replies
APJESK
by New Contributor III
  • 196 Views
  • 1 replies
  • 1 kudos

Clarification on Unity Catalog Metastore - Metadata and storage

Where does the Unity Catalog metastore metadata actually reside?Is it stored and managed in the Databricks account (control plane)?Or does it get stored in the customer-managed S3 bucket when we create a bucket for Unity Catalog metastore?I want to c...

  • 196 Views
  • 1 replies
  • 1 kudos
Latest Reply
nayan_wylde
Honored Contributor III
  • 1 kudos

@APJESK Replied here https://community.databricks.com/t5/data-governance/clarification-on-unity-catalog-metastore-metadata-and-storage/td-p/133389 

  • 1 kudos