Why is Spark creating multiple jobs for one action?

[Nastasia]

I noticed that when launching this bunch of code with only one action, I have three jobs that are launched.

from pyspark.sql import DataFrame from pyspark.sql.types import StructType, StructField, StringType from pyspark.sql.functions import avg

data: List = [("Diamant_1A", "TopDiamant", "300", "rouge"), ("Diamant_2B", "Diamants pour toujours", "45", "jaune"), ("Diamant_3C", "Mes diamants préférés", "78", "rouge"), ("Diamant_4D", "Diamants que j'aime", "90", "jaune"), ("Diamant_5E", "TopDiamant", "89", "bleu") ]

schema: StructType = StructType([ \ StructField("reference", StringType(), True), \ StructField("marque", StringType(), True), \ StructField("prix", StringType(), True), \ StructField("couleur", StringType(), True) ])

dataframe: DataFrame = spark.createDataFrame(data=data,schema=schema)

dataframe_filtree:DataFrame = dataframe.filter("prix > 50")

dataframe_filtree.show()

From my understanding, I should get only one. One action corresponds to one job.

I don't know why I have 3 jobs.

Here is the first one :

 

Here is the second one :

And this is the last one :

[Kaniz]

Hi @ Nastasia! My name is Kaniz, and I'm a technical moderator here. Great to meet you, and thanks for your question! Let's see if your peers on the Forum have an answer to your questions first. Or else I will follow up shortly with a response.

[chrishj]

I am very curious about this. Any answer?

[RKNutalapati]

The above code will create two jobs.

JOB-1. dataframe: DataFrame = spark.createDataFrame(data=data,schema=schema)

The createDataFrame function is responsible for inferring the schema from the provided data or using the specified schema.Depending on the data source, this might involve reading a small sample of the data to infer the schema correctly.This operation might be triggered lazily but can sometimes cause Spark to execute certain tasks immediately.

So, in practical terms, while createDataFrame is usually considered a transformation, there might be scenarios,
especially with certain data sources, where it involves internal actions for schema inference

JOB-2. dataframe_filtree.show()