Hi @aniket07 

With Serverless compute, Spark uses lazy evaluation and only checks if the path exists when you perform an action (like display()), so the error appears then. On the other hand, in All-Purpose clusters, Spark checks the path immediately when you create the DataFrame, so you see the error right away.

This difference is due to how each environment handles path validation and when they access storage.

Based on the scenario, what https://community.databricks.com/t5/user/viewprofilepage/user-id/156441 saying is correct though the eager evaluation property is false in both cases and for All-Purpose clusters, Spark is checking the path immediately when you create the DataFrame.