cancel
Turn on suggestions
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
Leodatabricks
by • Contributor
  • 7275 Views
  • 14 replies
  • 23 kudos

How to secure all clusters and then start running the code

When there are slow nodes, sometimes a job needs to resize its number of clusters to reach the required number of nodes. Is there any way to make sure no code is running before all nodes are secured? Thank you!

Data Engineering
Reply
  • 7275 Views
  • 14 replies
  • 23 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 23 kudos

Hi @Leo Bao​ Hope everything is going great.Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tell us so we ca...

  • 23 kudos
13 More Replies
uzairm
by • New Contributor III
  • 16106 Views
  • 2 replies
  • 2 kudos

Resolved! ThreadPoolExecutor in Databricks

I am using a threadpool executor and running notebooks in parallel. However, these parallel notebooks are not using executors at all and all the load is going towards the driver node resulting in running out of memory for the driver node and eventual...

Data Engineering
Reply
  • 16106 Views
  • 2 replies
  • 2 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 2 kudos

Hi @uzair mustafa​ Thank you for your question! To assist you better, please take a moment to review the answer and let me know if it best fits your needs.Please help us select the best solution by clicking on "Select As Best" if it does.Your feedbac...

  • 2 kudos
1 More Replies
MoJaMa
by • Databricks Employee
  • 6544 Views
  • 3 replies
  • 1 kudos

Does Databricks charge anything when nodes in a pool are idle?

Data Engineering
Reply
  • 6544 Views
  • 3 replies
  • 1 kudos
View Replies
Latest Reply
User16783853906
Contributor III
  • 1 kudos

Databricks does not charge DBUs while instances are idle in the pool. Instance provider billing does apply.Please refer here for more information - https://docs.databricks.com/clusters/instance-pools/index.html

  • 1 kudos
2 More Replies
Labels
Top