- 560 Views
- 3 replies
- 5 kudos
Resolved! dbt+Databrics
Hi!I will use dbt + Databrics in my new project.I'm runing dbt training. Have Databrics free/trial account so far. Want to connect, link dbt with Databbrics and create data warehouse from below data setsjaffle_shop_customers.csvjaffle_shop_orders.csv...
- 560 Views
- 3 replies
- 5 kudos
- 5 kudos
Great solution @szymon_dybczak .I don't know too much about DBT yet. Is there much difference between connecting/interacting with DBT cloud vs DBT core?All the best,BS
- 5 kudos
- 577 Views
- 3 replies
- 2 kudos
Transitioning Approach for Evolving EDW
As EDW will continue to evolve with new data and business logic during the multi-phased migration, what architectural strategies and design patterns can minimize rework when migrating from an evolving Enterprise Data Warehouse (EDW) to Databricks? I ...
- 577 Views
- 3 replies
- 2 kudos
- 2 kudos
There is no single approach. It depends on your organization.First you have the 'impact-axis' which is lean and mean vs big bang.Next you also have bottom-up (first fix bronze and work upwards) or top-down (focus on gold and read data from your legac...
- 2 kudos
- 1766 Views
- 6 replies
- 1 kudos
Resolved! Payment receipts of Databricks payments
Hello experts,I am trying to get receipts for the monthly payments done to Databricks. I need them for the financial department of the organization I am working for. The only billing information I get access to is the usage dashboards and the tables ...
- 1766 Views
- 6 replies
- 1 kudos
- 1 kudos
Hello everyone!I'd like to know how I can resolve a payment issue with Databricks?My credit card is registered to be charged automatically, but it hasn't been charged, as I received a message from Databricks asking me to make the payment, etc.Can any...
- 1 kudos
- 3858 Views
- 1 replies
- 0 kudos
Resolved! Ray cannot detect GPU on the cluster
I am trying to run ray on databricks for chunking and embedding tasks. The cluster I’m using is:g4dn.xlarge1-4 workers with 4-16 cores1 GPU and 16GB memoryI have set spark.task.resource.gpu.amount to 0.5 currently.This is how I have setup my ray clus...
- 3858 Views
- 1 replies
- 0 kudos
- 0 kudos
I have replicated all your steps and created the ray cluster exactly as you have done. Also, I have set: spark.conf.set("spark.task.resource.gpu.amount", "0.5") And I see a warning that shows that I don't allocate any GPU for Spark (as 1), even tho...
- 0 kudos
- 1078 Views
- 4 replies
- 2 kudos
Oauth Token federation
Dear allHas anyone tried oauth token federation for authentication with Databricks REST APIs?appreciate if there is a re-usable code snippet to achieve the same.
- 1078 Views
- 4 replies
- 2 kudos
- 2 kudos
@noorbasha534 Here is a sample python code I use for getting oauth token from Azure Active Directory and then pass the token in databricks API. Prerequisite is the SPN needs to be a admin in the workspace.import requests # Azure AD credentials tena...
- 2 kudos
- 375 Views
- 2 replies
- 1 kudos
Resolved! SQLSTATE HY000 after upgrading from Databricks 15.4 to 16.4
After upgrading from Databricks 15.4 to 16.4, without changing our Python code, we suddenly get SQL Timeouts, see below.Is there some new timeout default, that we don't know about, that we need to increase with the new version? After a quick search I...
- 375 Views
- 2 replies
- 1 kudos
- 1 kudos
After upgrading to Databricks 16.4, there is a notable change in SQL timeout behavior. The default timeout for SQL statements and objects like materialized views and streaming tables is now set to two days (172,800 seconds). This system-wide default ...
- 1 kudos
- 455 Views
- 2 replies
- 0 kudos
View Refresh Frequency
Dear allwe have around 5000+ finished data products (aka views) in several schemas of unity catalog. One question that comes from business users frequently is - how frequently these get refreshed?for that the answer is not simpler as the underlying t...
- 455 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @noorbasha534 just a pseudocode:for view in all_views:lineage = get_lineage(view) # Use Unity Catalog APIbase_tables = extract_base_tables(lineage)refresh_times = []for table in base_tables:job = find_job_refreshing_table(table) # Custom logic/met...
- 0 kudos
- 12991 Views
- 12 replies
- 7 kudos
Resolved! Unable to view Manage Account Option in Databricks UI
Hi All, I want to remove Unity Catalog from Admin Console so that I need to navigate to manage account option but the option is not available in my Databricks workspace. Please help me to sort this issue by removing Unity Catalog.
- 12991 Views
- 12 replies
- 7 kudos
- 662 Views
- 3 replies
- 2 kudos
Databricks OAuth: User-based OAuth (U2M) Databricks Connect in Apps
I'm looking to use a databricks session in a Databricks app. The databricks session should be able to use user-based oauth (U2M) to ensure the app has same privileges as the authenticated user using the app. Databricks apps have the ability to use th...
- 662 Views
- 3 replies
- 2 kudos
- 2 kudos
Thanks for you response and the links. But the documentation doesn't explicitly explain why the spark connect has been placed out of scope and what app builders should use to implement proper data governance using on behave of user permissions.
- 2 kudos
- 1931 Views
- 5 replies
- 3 kudos
Resolved! Connecting Azure databricks with firewall enabled Azure storage account
Hi I am trying to connect from Azure Databrick workspace to Azure gen2 storage account securely. The storage account is set up with these options1. Enabled from selected virtual networks and IP addresses- we whitelisted few ips 2. Added Microsoft.Dat...
- 1931 Views
- 5 replies
- 3 kudos
- 3 kudos
I am having exact issue as @trailblazer , that if I enable traffic for all network, I can read/write to storage account, if I only allow selected network, including the VNet, then it doesn't. I am using Serverless setup. I also followed the firewall ...
- 3 kudos
- 638 Views
- 1 replies
- 1 kudos
Resolved! Job Notifications specifically on Succeeded with Failures
Hi everyone,I have a set of jobs that always execute the last task regardless of whether the previous ones failed or not (using the ‘ALL done’ execution dependency).When moving to production and wanting to enable notifications, there is no option to ...
- 638 Views
- 1 replies
- 1 kudos
- 1 kudos
Databricks does not provide a direct way to distinguish or send notifications specifically for a "Succeeded with failures" state at the job level—the job is classified as "Success" even when some upstream tasks have failed, if the last (leaf) task is...
- 1 kudos
- 615 Views
- 2 replies
- 3 kudos
Resolved! Error when trying to destory databricks_permissions with OpenTofu
Hi,In our company's project we created a databricks_user for a service account (which is needed for our deployment process) via OpenTofu and afterwards adjusted permissions to that "user's" user folder using the databricks_permissions resource.resour...
- 615 Views
- 2 replies
- 3 kudos
- 3 kudos
Hi @MiriamHundemer , The issue occurs because the owner of the home folder (in this case, the databricks_user.databricks_deployment_sa service account) often has an unremovable CAN_MANAGE permission on its own home directory. When OpenTofu attempts t...
- 3 kudos
- 413 Views
- 4 replies
- 1 kudos
Resolved! Deply databricks workspace on azure with terraform - failed state: legacy access
I'm trying to deploy a workspace on azure via terraform and i'm getting the following error:"INVALID_PARAMETER_VALUE: Given value cannot be set for workspace~<id>~default_namespace_ws~ because: cannot set default namespace to hive_metastore since leg...
- 413 Views
- 4 replies
- 1 kudos
- 1 kudos
I found the issue, The setting automatically assigned workspaces to this metastore was checked. Unchecking this and manually assigning the metastore worked.
- 1 kudos
- 246 Views
- 1 replies
- 1 kudos
Clarification on Unity Catalog Metastore - Metadata and storage
Where does the Unity Catalog metastore metadata actually reside?Is it stored and managed in the Databricks account (control plane)?Or does it get stored in the customer-managed S3 bucket when we create a bucket for Unity Catalog metastore?I want to c...
- 246 Views
- 1 replies
- 1 kudos
- 1 kudos
@APJESK Replied here https://community.databricks.com/t5/data-governance/clarification-on-unity-catalog-metastore-metadata-and-storage/td-p/133389
- 1 kudos
- 382 Views
- 3 replies
- 3 kudos
Resolved! Databricks GCP login with company account
I created gmail on my company account, then I logged in in gcp databricks, till yesterday it was working fine, yesterday I logged in into gmail account, there it asked for another gmail id, so I have provided new, but today I am not able to login usi...
- 382 Views
- 3 replies
- 3 kudos
- 3 kudos
Hello @xavier_db! Were you able to get this login issue resolved? If yes, it would be great if you could share what worked for you so others facing the same problem can benefit as well.
- 3 kudos
