cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
uzairm
by • Databricks Partner
  • 21553 Views
  • 4 replies
  • 2 kudos

Resolved! ThreadPoolExecutor in Databricks

I am using a threadpool executor and running notebooks in parallel. However, these parallel notebooks are not using executors at all and all the load is going towards the driver node resulting in running out of memory for the driver node and eventual...

Data Engineering
Reply
  • 21553 Views
  • 4 replies
  • 2 kudos
View Replies
Latest Reply
srimonishan
New Contributor
  • 2 kudos

Good point on the threading vs. process distinction. To add a bit morecontext:Root cause: ThreadPoolExecutor only parallelizes within the driver'ssingle Python process. All threads share the same driver memory, sopandas DataFrame creation across thre...

  • 2 kudos
3 More Replies
Leodatabricks
by • Contributor
  • 12478 Views
  • 14 replies
  • 23 kudos

How to secure all clusters and then start running the code

When there are slow nodes, sometimes a job needs to resize its number of clusters to reach the required number of nodes. Is there any way to make sure no code is running before all nodes are secured? Thank you!

Data Engineering
Reply
  • 12478 Views
  • 14 replies
  • 23 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 23 kudos

Hi @Leo Bao​ Hope everything is going great.Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tell us so we ca...

  • 23 kudos
13 More Replies
MoJaMa
by • Databricks Employee
  • 13623 Views
  • 3 replies
  • 1 kudos

Does Databricks charge anything when nodes in a pool are idle?

Data Engineering
Reply
  • 13623 Views
  • 3 replies
  • 1 kudos
View Replies
Latest Reply
User16783853906
Databricks Employee
  • 1 kudos

Databricks does not charge DBUs while instances are idle in the pool. Instance provider billing does apply.Please refer here for more information - https://docs.databricks.com/clusters/instance-pools/index.html

  • 1 kudos
2 More Replies
Labels