topic Re: Error when executing spark.readStream Script in Get Started Discussions

Error when executing spark.readStream Script

giuseppe_esq — Mon, 25 Aug 2025 20:39:11 GMT

Hi all,

When I try to execute the script (as per the screenshot below), in a Notebook cell, I get an error message. I am using Databricks Free Edition, and am not sure if the error relates to the compute cluster that I am using?

Any guidance would be greatly appreciated.

Thanks

Giuseppe

Re: Error when executing spark.readStream Script

nayan_wylde — Mon, 25 Aug 2025 20:45:06 GMT

@giuseppe_esq

Can you please copy the entire error message and paste here.

Re: Error when executing spark.readStream Script

giuseppe_esq — Mon, 25 Aug 2025 20:53:13 GMT

Yep, as requested.

Thanks

Re: Error when executing spark.readStream Script

brycejune — Tue, 26 Aug 2025 09:17:48 GMT

Free Edition clusters often come with restricted resources and certain features may not be supported. You may want to review your cluster settings, check compatibility with your script, and consider whether upgrading to a paid tier is necessary for your use case.

Re: Error when executing spark.readStream Script

Sidhant07 — Tue, 26 Aug 2025 10:43:52 GMT

Hi @giuseppe_esq ,

Can you please share the DBR version, cluster configurations and log4 file containing the error stact trace to further review?

Re: Error when executing spark.readStream Script

giuseppe_esq — Tue, 26 Aug 2025 16:01:55 GMT

Hi,

Thanks for your reply.

Apologies if I am sending the incorrect information. I am in the process of learning databricks, and adding content to my personal blog.

DBR Version

Cluster Configurations

From the Compute menu, this is the only cluster available in the Databricks Free Edition:

From the Workspace Notebook, I also tried to attach the General Compute option, though this didn't resolve the issue either:

Log4j File

I have attached the Log4j File, for the time I executed the following query, that errored:

## Incrementally (or stream) data using Auto Loader
(spark.
readStream
.format("cloudFiles")
.option("cloudFiles.format", "csv")
.option("header", "true")
.option("sep", ",")
.option("inferSchema", "true")
.option("cloudFiles.schemaLocation", f"{checkpoint_file_location}")
.load("/Volumes/workspace/python_auto_loader/csv_staging")
.writeStream
.option("checkpointLocation", f"{checkpoint_file_location}")
.trigger(once=True)
.toTable("workspace.python_auto_loader.python_csv_autoloader")
)

If you require any additional information, please message me again.

Thank you for your help.

Giuseppe

Re: Error when executing spark.readStream Script

giuseppe_esq — Sat, 06 Sep 2025 13:36:00 GMT

Hi Sidhant,

Are there any updates on this please?

Thanks for your time and help

Giuseppe