Hello community,

I use a cluster (Single node: Standard_F64s_v2 · DBR: 14.3 LTS (includes Apache Spark 3.5.0, Scala 2.12)) for a job. In this job I didn't use spark multiprocessing. Instead I use this single node cluster as a VM and use python multiprocessing to finish my job. 

Before 2024-08-06 04:30 (CET) I can use the CPU of the single node cluster relative full. And the job can be finished less than one hour. But after 2024-08-06 04:30 (CET) the CPU of the single node cluster can not be fully used, it remains under 25%. Even though I didn't make any changes.

So does anyone know why this is happening? Is there any changes made around 2024-08-06 04:30 (CET) on Data Bricks? some update or something like that?

Thanks for any advice and really appreciate for any help.

Best regards,

Narsu