cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

Why is Spark creating multiple jobs for one action?

Nastasia
New Contributor II

I noticed that when launching this bunch of code with only one action, I have three jobs that are launched.

from pyspark.sql import DataFrame from pyspark.sql.types import StructType, StructField, StringType from pyspark.sql.functions import avg
data: List = [("Diamant_1A", "TopDiamant", "300", "rouge"), ("Diamant_2B", "Diamants pour toujours", "45", "jaune"), ("Diamant_3C", "Mes diamants prรฉfรฉrรฉs", "78", "rouge"), ("Diamant_4D", "Diamants que j'aime", "90", "jaune"), ("Diamant_5E", "TopDiamant", "89", "bleu") ]
schema: StructType = StructType([ \ StructField("reference", StringType(), True), \ StructField("marque", StringType(), True), \ StructField("prix", StringType(), True), \ StructField("couleur", StringType(), True) ])
dataframe: DataFrame = spark.createDataFrame(data=data,schema=schema)
dataframe_filtree:DataFrame = dataframe.filter("prix > 50")
dataframe_filtree.show()

From my understanding, I should get only one. One action corresponds to one job.

I don't know why I have 3 jobs.

Here is the first one :

https___i.stack.imgur.com_xfYDe.png 

Here is the second one :

LTHBMAnd this is the last one :

DdfHN

2 REPLIES 2

chrishj
New Contributor II

I am very curious about this. Any answer?

RKNutalapati
Valued Contributor

The above code will create two jobs.

JOB-1. dataframe: DataFrame = spark.createDataFrame(data=data,schema=schema)

The createDataFrame function is responsible for inferring the schema from the provided data or using the specified schema.Depending on the data source, this might involve reading a small sample of the data to infer the schema correctly.This operation might be triggered lazily but can sometimes cause Spark to execute certain tasks immediately.

So, in practical terms, while createDataFrame is usually considered a transformation, there might be scenarios,
especially with certain data sources, where it involves internal actions for schema inference

JOB-2. dataframe_filtree.show()

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโ€™t want to miss the chance to attend and share knowledge.

If there isnโ€™t a group near you, start one and help create a community that brings people together.

Request a New Group