topic Re: Databricks Issue:- assertion failed: Invalid shuffle partition specs: in Data Engineering

Databricks Issue:- assertion failed: Invalid shuffle partition specs:

KumarShiv — Wed, 27 Jul 2022 07:41:49 GMT

I hv a complex script which consuming more then 100GB data and have some aggregation on it and in the end I am simply try simply write/display data from Data frame. Then i am getting issue (assertion failed: Invalid shuffle partition specs: ).

Pls help me here, if any one have some idea.

Re: Databricks Issue:- assertion failed: Invalid shuffle partition specs:

Hubert-Dudek — Wed, 27 Jul 2022 10:42:36 GMT

It is hard to help in that case without seeing the whole code.

Re: Databricks Issue:- assertion failed: Invalid shuffle partition specs:

KumarShiv — Wed, 27 Jul 2022 11:02:23 GMT

Added ".py" file in attachment, Pls have a look.

Re: Databricks Issue:- assertion failed: Invalid shuffle partition specs:

Hubert-Dudek — Wed, 27 Jul 2022 13:10:11 GMT

Please use

display(df_FinalAction)

Spark is lazy evaluated but "display" not, so you can debug by displaying each dataframe at the end of each cell.

Re: Databricks Issue:- assertion failed: Invalid shuffle partition specs:

KumarShiv — Thu, 28 Jul 2022 14:54:43 GMT

Thanks Dudek, Finally I found the query where that have issue. As you suggested I debug step by step and run each every cell. And add the (spark.conf.set("spark.sql.shuffle.partitions",100)) at that cell. Its resolved 😀

Re: Databricks Issue:- assertion failed: Invalid shuffle partition specs:

Hubert-Dudek — Fri, 29 Jul 2022 11:10:11 GMT

Thanks, it is excellent. If you want you can select my answer as the best one 🙂