topic Re: Serverless compute vs Job cluster in Data Engineering

Serverless compute vs Job cluster

vaibhavaher2025 — Fri, 21 Feb 2025 15:13:11 GMT

Hi Guys,
For running the job with varying workload what should I use ? Serverless cluster or Job compute ?
What are positives and negatives?

(I'll be running my notebook from Azure data factory)

Re: Serverless compute vs Job cluster

MariuszK — Fri, 21 Feb 2025 16:33:06 GMT

Hi
If you use PySpark for data processing than Job compute it has:
- lower cost
- support for PySpark
- flexible configuration
cons:
- Slower startup time

Serverless Warehouse:
- Faster startup time
- Dedicated for SQL
- Lower management
cons:
- not supporting PySpark
- more expensive for compute unit (Photon acceleration)
- less customization

Re: Serverless compute vs Job cluster

KaranamS — Fri, 21 Feb 2025 17:33:37 GMT

It depends on cost, performance and startup time needed for your use-case.

Serverless compute is usually preferred choice because of its fast startup time and dynamic scaling. However, if your workload is long-running and predictable, job compute with auto scaling might be more cost-effective.