topic Re: Why is Spark creating multiple jobs for one action? in Data Engineering

Why is Spark creating multiple jobs for one action?

Nastasia — Thu, 29 Jul 2021 12:33:27 GMT

I noticed that when launching this bunch of code with only one action, I have three jobs that are launched.

from pyspark.sql import DataFrame from pyspark.sql.types import StructType, StructField, StringType from pyspark.sql.functions import avg

data: List = [("Diamant_1A", "TopDiamant", "300", "rouge"), ("Diamant_2B", "Diamants pour toujours", "45", "jaune"), ("Diamant_3C", "Mes diamants préférés", "78", "rouge"), ("Diamant_4D", "Diamants que j'aime", "90", "jaune"), ("Diamant_5E", "TopDiamant", "89", "bleu") ]

schema: StructType = StructType([ \ StructField("reference", StringType(), True), \ StructField("marque", StringType(), True), \ StructField("prix", StringType(), True), \ StructField("couleur", StringType(), True) ])

dataframe: DataFrame = spark.createDataFrame(data=data,schema=schema)

dataframe_filtree:DataFrame = dataframe.filter("prix > 50")

dataframe_filtree.show()

From my understanding, I should get only one. One action corresponds to one job.

I don't know why I have 3 jobs.

Here is the first one :

Here is the second one :

And this is the last one :

Re: Why is Spark creating multiple jobs for one action?

chrishj — Mon, 18 Dec 2023 17:34:05 GMT

I am very curious about this. Any answer?

Re: Why is Spark creating multiple jobs for one action?

RKNutalapati — Tue, 19 Dec 2023 21:33:53 GMT

The above code will create two jobs.

JOB-1. dataframe: DataFrame = spark.createDataFrame(data=data,schema=schema)

The createDataFrame function is responsible for inferring the schema from the provided data or using the specified schema.Depending on the data source, this might involve reading a small sample of the data to infer the schema correctly.This operation might be triggered lazily but can sometimes cause Spark to execute certain tasks immediately.

So, in practical terms, while createDataFrame is usually considered a transformation, there might be scenarios,
especially with certain data sources, where it involves internal actions for schema inference

JOB-2. dataframe_filtree.show()