Databricks Issue:- assertion failed: Invalid shuffle partition specs:

[KumarShiv]

I hv a complex script which consuming more then 100GB data and have some aggregation on it and in the end I am simply try simply write/display data from Data frame. Then i am getting issue (assertion failed: Invalid shuffle partition specs: ).

Pls help me here, if any one have some idea.

[Hubert-Dudek]

It is hard to help in that case without seeing the whole code.

[KumarShiv]

Added ".py" file in attachment, Pls have a look.

[Hubert-Dudek]

Please use

display(df_FinalAction)

Spark is lazy evaluated but "display" not, so you can debug by displaying each dataframe at the end of each cell.

[KumarShiv · Accepted solution]

Thanks Dudek, Finally I found the query where that have issue. As you suggested I debug step by step and run each every cell. And add the (spark.conf.set("spark.sql.shuffle.partitions",100)) at that cell. Its resolved 😀

[Hubert-Dudek]

Thanks, it is excellent. If you want you can select my answer as the best one 🙂