More than expected number of Jobs created in Databricks
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-02-2023 03:45 AM
Hi Databricks Gurus !
I am trying to run a very simple snippet :
data_emp=[["1","sarvan","1"],["2","John","2"],["3","Jose","1"]]
emp_columns=["EmpId","Name","Dept"]
df=spark.createDataFrame(data=data_emp, schema=emp_columns)
df.show()
--------
Based on a general understanding data bricks should create at the most 2 Jobs
One to read the data(this works for files like that, don't know if it would apply here)
One for show()
But it somehow creating 3 jobs
Can someone explain why is the behavior ?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-03-2023 04:00 PM
Thank You @Retired_mod !!
I was also suspecting the same and your response helped in the conclusion
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-10-2025 10:30 AM
Hi @DBEnthusiast,
Please share or explain the solution as I am unable to see previous reply on this thread.