topic How to Create Cluster for running JOB in Databricks Free Edition Help

How to Create Cluster for running JOB

ShailenGenpact — Wed, 12 Mar 2025 17:55:21 GMT

I am encountering an error message during the executing a job (workflow-pipiline) with computing serverless as by default but it required Single node cluster and I am not able to see any option to create cluster in my subscription even I added my credit card and I have $390 credit as balance.

Currently I am facing an issue while running a job (workflow-pipiline).

"Directly accessing the underlying Spark driver JVM using the attribute 'sparkContext' is not supported on serverless compute. If you require direct access to these fields, consider using a single-user cluster. For more details on compatibility and limitations, check: https://docs.databricks.com/release-notes/serverless.html#limitationsFile <command-3107834473466653>, line 2"

Screenshot 2025-03-12 124910

Re: How to Create Cluster for running JOB

Akash_Varuna — Thu, 13 Mar 2025 07:32:43 GMT

Hi @ShailenGenpact Could you check this post maybe this might help you
https://community.databricks.com/t5/data-engineering/sparkcontext-in-runtime-15-3/td-p/81646
In this they have said similar case and some alternate workarounds they have used

Re: How to Create Cluster for running JOB

saisaran_g — Fri, 04 Apr 2025 17:31:40 GMT

Hello Mate,

Did you got the issue fixed ?

Serverless will not support on few elements and having some limitations, instead you can configure the single node cluster in the workflows section itself will fix the issue.

Thanks for the ask, happy learning.

Saran