Databricks Community

KumarShiv · 08-01-2022

I am try to get the percentile values on different splits but I got that the result of Databricks PERCENTILE_DISC() function is not accurate . I have run the same query on MS SQL but getting different result set.Here are both result sets for Pyspark ...

KumarShiv · 07-27-2022

I hv a complex script which consuming more then 100GB data and have some aggregation on it and in the end I am simply try simply write/display data from Data frame. Then i am getting issue (assertion failed: Invalid shuffle partition specs: ).Pls hel...

KumarShiv · 07-28-2022

Thanks Dudek, Finally I found the query where that have issue. As you suggested I debug step by step and run each every cell. And add the (spark.conf.set("spark.sql.shuffle.partitions",100)) at that cell. Its resolved 

KumarShiv · 07-27-2022

Added ".py" file in attachment, Pls have a look.

Databricks Community

User Stats

User Activity

Databricks Spark SQL function "PERCENTILE_DISC()" output not accurate.

Databricks Issue:- assertion failed: Invalid shuffle partition specs:

Re: Databricks Issue:- assertion failed: Invalid shuffle partition specs:

Re: Databricks Issue:- assertion failed: Invalid shuffle partition specs: