PySpark Lazy Evaluation

joggiri — Thu, 07 Aug 2025 16:02:42 GMT

PySpark Lazy Evaluation - Why does my logging function seem to execute without an explicit action in Databricks?

Hello everyone,
I was scrolling and found some Medium post on a PySpark (https://medium.com/@sudeepwrites/pyspark-secrets-no-one-talks-about-but-every-data-engineer-should-know-876a0202864a) and have a question about lazy evaluation. It is written that transformations are lazy and will not execute until an action is called (which I know).
Arctical have some code I have tried to execute, according to article it should not print 'Logging something...' ,However, It is printing.

from pyspark.sql.functions import coldef log_step(df):
print("Logging something...")
return df

df = spark.sql("select * from delta.`s3://abc` limit 10")
log_step(df.filter(col("flag_source_system_delete") == "yes"))

Even without an explicit action like .show() or .count() on the final line, the print() statement inside log_step executes and I see the output in my notebook. My understanding is that the filter is a transformation and should not trigger the code.
Can someone please explain why this is happening? Is there an implicit action being triggered by the Databricks notebook environment, or am I fundamentally misunderstanding something about lazy evaluation with functions?

Thank you!

Re: PySpark Lazy Evaluation

cgrant — Fri, 08 Aug 2025 16:16:31 GMT

I don't have full access to that article, but here's something that might help clarify things!

While Spark uses lazy evaluation (meaning it waits to execute until absolutely necessary), Python works with eager evaluation. This means that when you run log_step, the Python code jumps into action and executes right away

topic PySpark Lazy Evaluation in Data Engineering

PySpark Lazy Evaluation

Re: PySpark Lazy Evaluation