RKNutalapati
Valued Contributor

The above code will create two jobs.

JOB-1. dataframe: DataFrame = spark.createDataFrame(data=data,schema=schema)

The createDataFrame function is responsible for inferring the schema from the provided data or using the specified schema.Depending on the data source, this might involve reading a small sample of the data to infer the schema correctly.This operation might be triggered lazily but can sometimes cause Spark to execute certain tasks immediately.

So, in practical terms, while createDataFrame is usually considered a transformation, there might be scenarios,
especially with certain data sources, where it involves internal actions for schema inference

JOB-2. dataframe_filtree.show()