Databricks

DennisB · 07-24-2023

Hi everyone,Hoping someone can help me with this problem. I have an embarrassingly parallel workload, which I'm parallelising over 4 worker nodes (of type Standard_F4, so 4 cores each). Each workload is single-threaded, so I believe that only one cor...

DennisB · 03-14-2024

I was getting a similar error (full traceback below), and determined that it's related to this issue. Setting the env variables DATABRICKS_HOST and DATABRICKS_TOKEN as suggested in that Github issue resolved the problem for me (albeit it's not a grea...

DennisB · 07-26-2023

I haven't tried threadpool, but thanks for the suggestion. Before posting I had tried a multiprocessing Pool, but that didn't work (I'd hoped to use multiprocessing on each worker now, i.e., Spark to distribute to worker nodes, then multiprocessing t...

DennisB · 07-26-2023

So I managed to get the 1-core-per-executor working successfully. The bit that wasn't working was spark.executor.memory -- this was too high, but lowering it so that the sum of the executors memory was ~90% of the worker node's memory allowed it to w...

Databricks

User Stats

User Activity

Better Worker Node Core Utilisation

Re: UDF importing from other modules

Re: Better Worker Node Core Utilisation

Re: Better Worker Node Core Utilisation