Prajapathy_NKR
Contributor

@Hritik_Moon 

1. check if your data is cached, this you can see in sparkUI > storage tab.

2. if it is not cached, try to add a action statement after you cache. eg : df.count(). Data is cached with the first action statement it encounters. Now check in spark UI.

3. if you have only one action statement, you dont see any difference. But if you have multiple action statement, you tend to see the relevant transformations before your cached dataframe gets skipped. You can see these skips in your DAG.