single node Cluster CPU not fully used

narisuna — Thu, 08 Aug 2024 09:42:25 GMT

Hello community,

I use a cluster (Single node: Standard_F64s_v2 · DBR: 14.3 LTS (includes Apache Spark 3.5.0, Scala 2.12)) for a job. In this job I didn't use spark multiprocessing. Instead I use this single node cluster as a VM and use python multiprocessing to finish my job.

Before 2024-08-06 04:30 (CET) I can use the CPU of the single node cluster relative full. And the job can be finished less than one hour. But after 2024-08-06 04:30 (CET) the CPU of the single node cluster can not be fully used, it remains under 25%. Even though I didn't make any changes.

So does anyone know why this is happening? Is there any changes made around 2024-08-06 04:30 (CET) on Data Bricks? some update or something like that?

Thanks for any advice and really appreciate for any help.

Best regards,

Narsu

Re: single node Cluster CPU not fully used

Sidhant07 — Mon, 09 Dec 2024 08:07:52 GMT

Hi,

Yes there was a maintainance release on 7th august which might have caused this issue.

If you are still experience this issue, please file a support ticket.

topic Re: single node Cluster CPU not fully used in Data Engineering

single node Cluster CPU not fully used

Re: single node Cluster CPU not fully used