cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
278875
by • New Contributor
  • 7402 Views
  • 4 replies
  • 1 kudos

How do I figure out the cost breakdown for Databricks

I'm trying to figure out the cost breakdown for the Databricks usage for my team.When I go into the Databricks administration console and click Usage when I select to show the usage By SKU it just displays the type of cluster but not the name of it. ...

Data Engineering
Reply
  • 7402 Views
  • 4 replies
  • 1 kudos
View Replies
Latest Reply
MuthuLakshmi
New Contributor III
  • 1 kudos

Please check the below docs for usage related informations. The Billable Usage Logs: https://docs.databricks.com/en/administration-guide/account-settings/usage.html You can filter them using tags for more precise information which you are looking for...

  • 1 kudos
3 More Replies
Tahseen0354
by • Valued Contributor
  • 8750 Views
  • 15 replies
  • 39 kudos

How do I compare cost between databricks gcp and azure databricks ?

I have a databricks job running in azure databricks. A similar job is also running in databricks gcp. I would like to compare the cost. If I assign a custom tag to the job cluster running in azure databricks, I can see the cost incurred by that job i...

Data Engineering
Reply
  • 8750 Views
  • 15 replies
  • 39 kudos
View Replies
Latest Reply
Own
Contributor
  • 39 kudos

In Azure, you can use Cost Management to track your expenses incurred by Databricks instance.

  • 39 kudos
14 More Replies
Karthikeyan1
by • New Contributor II
  • 940 Views
  • 1 replies
  • 2 kudos

How to get the Databricks Default Pricing through API or any endpoints (*if available)???

Attaching Screenshots FYI from the official site, I've checked the Inspect, but no API calls have been made specifically for this cost default scrapping, Is there any Endpoints available to scrape this?

image.png
Data Engineering
Reply
  • 940 Views
  • 1 replies
  • 2 kudos
View Replies
Latest Reply
Karthikeyan1
New Contributor II
  • 2 kudos

@All Users Group​ Any views on this??

  • 2 kudos
Laniel
by • New Contributor
  • 920 Views
  • 1 replies
  • 0 kudos

‘How do you get cost of a notebook run?’

‘How do you get cost of a notebook run?’

Data Engineering
Reply
  • 920 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
Rheiman
Contributor II
  • 0 kudos

You can check your cloud provider's portal. Go to the subscription > costs field and you should be able to see the costs of the VMs and Databricks. For more granular information, consider installing overwatch.Environment Setup :: Overwatch (databrick...

  • 0 kudos
dataslicer
by • Contributor
  • 4645 Views
  • 7 replies
  • 2 kudos

Resolved! Exploring additional cost saving options for structured streaming 24x7x365 uptime workloads

I currently have multiple jobs (each running its own job cluster) for my spark structured streaming pipelines that are long running 24x7x365 on DBR 9.x/10.x LTS. My SLAs are 24x7x365 with 1 minute latency. I have already accomplished the following co...

Data Engineering
Reply
  • 4645 Views
  • 7 replies
  • 2 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 2 kudos

http://doramasmp4.tv/

  • 2 kudos
6 More Replies
User16857281869
by • New Contributor II
  • 1925 Views
  • 1 replies
  • 1 kudos

Resolved! Why do I see a cost explosion in my blob storage account (DBFS storage, blob storage, ...) for my structures streaming job?

Its usually one or more of the following reasons:1) If you are streaming into a table, you should be using .Trigger option to specify the frequency of checkpointing. Otherwise, the job will call the storage API every 10ms to log the transaction data...

Data Engineering
Reply
  • 1925 Views
  • 1 replies
  • 1 kudos
View Replies
Latest Reply
Hubert-Dudek
Esteemed Contributor III
  • 1 kudos

please mount cheaper storage (LRS) to custom mount and set there checkpoints,please clear data regularly,if you are using forEac/forEatchBatchh in stream it will save every dataframe on dbfs,please remember not to use display() in production,if on th...

  • 1 kudos
Mohit_m
by • Valued Contributor II
  • 1936 Views
  • 2 replies
  • 1 kudos

Resolved! Why EC2 or EBS volumes are not getting tagged for the instances pool

Why EC2 or EBS volumes are not getting tagged for the instances pool when the tags are present for the Clusters

Data Engineering
Reply
  • 1936 Views
  • 2 replies
  • 1 kudos
View Replies
Latest Reply
Hubert-Dudek
Esteemed Contributor III
  • 1 kudos

"If a cluster is created from a pool, its EC2 instances inherit only the custom and default pool tags, not the cluster tags. Therefore if you want to create clusters from a pool, make sure to assign all of the custom cluster tags you need to the pool...

  • 1 kudos
1 More Replies
Labels