topic Re: Lazy evaluation in serverless vs all purpose compute ? in Get Started Discussions

Lazy evaluation in serverless vs all purpose compute ?

aniket07 — Tue, 15 Apr 2025 04:29:36 GMT

As you can see right now I am connected to serverless compute and when I give wrong path, spark does lazy evaluation and gives error on display.
However, when I switch from serverless to my all purpose cluster I get the error when I create the df itself.
Why is that?

Re: Lazy evaluation in serverless vs all purpose compute ?

SP_6721 — Tue, 15 Apr 2025 13:09:00 GMT

Hi @aniket07

With Serverless compute, Spark uses lazy evaluation and only checks if the path exists when you perform an action (like display()), so the error appears then. On the other hand, in All-Purpose clusters, Spark checks the path immediately when you create the DataFrame, so you see the error right away.

This difference is due to how each environment handles path validation and when they access storage.

Re: Lazy evaluation in serverless vs all purpose compute ?

sridharplv — Tue, 15 Apr 2025 18:01:10 GMT

Based on the scenario, what https://community.databricks.com/t5/user/viewprofilepage/user-id/156441 saying is correct though the eager evaluation property is false in both cases and for All-Purpose clusters, Spark is checking the path immediately when you create the DataFrame.