topic Unable to reproduce Kmeans Clustering results even after setting seed and tolerance in Data Engineering

Unable to reproduce Kmeans Clustering results even after setting seed and tolerance

mala — Tue, 10 Jan 2023 23:25:20 GMT

I have been trying to reproduce Kmeans results with no luck

Here is my code snippet:

from pyspark.ml.clustering import KMeans

KMeans(featuresCol=featuresCol, k=clusters, maxIter=40, seed=1, tol = .00001)

Can anyone help?

Re: Unable to reproduce Kmeans Clustering results even after setting seed and tolerance

mala — Wed, 11 Jan 2023 21:20:20 GMT

Hi Debaya

Thanks for your reply, it runs without any issues. After rerunning the model each time, I got different cluster outputs even after applying seed and tolerance as I have mentioned in my code snippet.

I would expect the results to be the same once you apply seed since it removes any randomness. I also increased the number of iterations which didn't help either.

Is there a way to reproduce the results in Spark?

Thanks

Mala

Re: Unable to reproduce Kmeans Clustering results even after setting seed and tolerance

mala — Thu, 19 Jan 2023 18:52:56 GMT

This issue was due to spark parallelization which doesn't guarantee the same data is assigned to each partition.

I was able to resolve this by making sure the same data is assigned to the same partitions :

df.repartition(num_partitions, "ur_col_id")

df.sortWithinPartitions("ur_col_id")

Re: Unable to reproduce Kmeans Clustering results even after setting seed and tolerance

Debayan — Wed, 11 Jan 2023 21:11:45 GMT

Hi, Do you receive any errors? Please refer https://www.databricks.com/tensorflow/clustering-and-k-means for examples. Please let us know if this helps.