12-02-2023 03:45 AM
Hi Databricks Gurus !
I am trying to run a very simple snippet :
data_emp=[["1","sarvan","1"],["2","John","2"],["3","Jose","1"]]
emp_columns=["EmpId","Name","Dept"]
df=spark.createDataFrame(data=data_emp, schema=emp_columns)
df.show()
--------
Based on a general understanding data bricks should create at the most 2 Jobs
One to read the data(this works for files like that, don't know if it would apply here)
One for show()
But it somehow creating 3 jobs
Can someone explain why is the behavior ?
12-03-2023 07:31 AM
Hi @DBEnthusiast, Hello! Let’s dive into the behaviour you’re observing with your simple Databricks snippet.
The code you provided creates a DataFrame named df using the createDataFrame method. Then, it displays the first few rows of the DataFrame using the show() method. However, you’ve noticed that it’s resulted in three jobs being created instead of the expected two.
Here’s what’s happening:
Job 1: DataFrame Creation
Job 2: Show Operation
Additional Job (Unexpected)
Why the Extra Job?
Recommendations:
Feel free to explore further or ask any additional questions! 😊
12-03-2023 07:31 AM
Hi @DBEnthusiast, Hello! Let’s dive into the behaviour you’re observing with your simple Databricks snippet.
The code you provided creates a DataFrame named df using the createDataFrame method. Then, it displays the first few rows of the DataFrame using the show() method. However, you’ve noticed that it’s resulted in three jobs being created instead of the expected two.
Here’s what’s happening:
Job 1: DataFrame Creation
Job 2: Show Operation
Additional Job (Unexpected)
Why the Extra Job?
Recommendations:
Feel free to explore further or ask any additional questions! 😊
12-03-2023 04:00 PM
Thank You @Kaniz_Fatma !!
I was also suspecting the same and your response helped in the conclusion
12-10-2023 11:25 PM
I want to express my gratitude for your effort in selecting the most suitable solution. It's great to hear that your query has been successfully resolved. Thank you for your contribution.
Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.
If there isn’t a group near you, start one and help create a community that brings people together.
Request a New Group