https://community.databricks.com/t5/data-engineering/quantilediscretizer-not-respecting-numbuckets/m-p/16030#M10255 <P>Thank you.</P><P></P><P>What I did was:</P><OL><LI>Apply QuntileBucketizer to Non-Zeros and specified a very small value (bottom 1%) to capture the lower bucket including zeroes.</LI></OL><P></P><P>That fixed the issue! You can define your own splits which would work as well but the splits themselves were important in this case.</P> Tue, 14 Sep 2021 01:19:17 GMT Sam 2021-09-14T01:19:17Z https://community.databricks.com/t5/data-engineering/quantilediscretizer-not-respecting-numbuckets/m-p/16027#M10252 <P>I have set numBuckets and <B>numBucketsArray</B> for a group of columns to bin them into 5 buckets.</P><P></P><P>Unfortunately the number of buckets does not seem to be respected across all columns even though there is variation within them.</P><P></P><P>I have tried setting the relativeerror to 0.</P><P></P><P><A href="https://spark.apache.org/docs/latest/api/python/reference/api/pyspark.ml.feature.QuantileDiscretizer.html" target="test_blank">https://spark.apache.org/docs/latest/api/python/reference/api/pyspark.ml.feature.QuantileDiscretizer.html</A></P><P></P><P>Any idea why this is and how to solve it to force the number of buckets specified?</P> Thu, 02 Sep 2021 22:39:48 GMT https://community.databricks.com/t5/data-engineering/quantilediscretizer-not-respecting-numbuckets/m-p/16027#M10252 Sam 2021-09-02T22:39:48Z https://community.databricks.com/t5/data-engineering/quantilediscretizer-not-respecting-numbuckets/m-p/16029#M10254 <P>QuantileDiscretizer does not guarantee the number of buckets afaik. Depending on your data you might get less buckets than asked.</P><P>Bucketizer however does, but you have to define your splits.</P> Fri, 03 Sep 2021 13:11:52 GMT https://community.databricks.com/t5/data-engineering/quantilediscretizer-not-respecting-numbuckets/m-p/16029#M10254 -werners- 2021-09-03T13:11:52Z https://community.databricks.com/t5/data-engineering/quantilediscretizer-not-respecting-numbuckets/m-p/16030#M10255 <P>Thank you.</P><P></P><P>What I did was:</P><OL><LI>Apply QuntileBucketizer to Non-Zeros and specified a very small value (bottom 1%) to capture the lower bucket including zeroes.</LI></OL><P></P><P>That fixed the issue! You can define your own splits which would work as well but the splits themselves were important in this case.</P> Tue, 14 Sep 2021 01:19:17 GMT https://community.databricks.com/t5/data-engineering/quantilediscretizer-not-respecting-numbuckets/m-p/16030#M10255 Sam 2021-09-14T01:19:17Z https://community.databricks.com/t5/data-engineering/quantilediscretizer-not-respecting-numbuckets/m-p/16031#M10256 <P>Can you explain a bit more?​</P> Thu, 14 Jul 2022 03:13:08 GMT https://community.databricks.com/t5/data-engineering/quantilediscretizer-not-respecting-numbuckets/m-p/16031#M10256 Hemant 2022-07-14T03:13:08Z