- 1522 Views
- 1 replies
- 1 kudos
Error on pandas udf usage in databricks, sc.broadcasting random forest loaded from Kedro MLFlow Logger DataSet, cannot pickle '_thread.RLock' object
I'm trying to broadcast a Random forest (sklearn 1.2.0) recently loaded from mlflow, and using Pandas UDF to predict a model.​However, the same code works perfectly on Spark 2.4 + our OnPrem cluster.​I thought it was due to Spark 2.4 to 3 changes, an...
- 1522 Views
- 1 replies
- 1 kudos
- 1 kudos
Anyone?
- 1 kudos
- 2742 Views
- 4 replies
- 2 kudos
How to solve cluster break down due to GC when training a pyspark.ml Random Forest
I am trying to train and optimize a random forest. At first the cluster handles the garbage collection fine, but after a couple of hours the cluster breaks down as Garbage Collection has gone up significantly.The train_df has a size of 6,365,018 reco...
- 2742 Views
- 4 replies
- 2 kudos
- 2 kudos
Hi @Liselotte van Unen​(Customer)​ , We haven’t heard from you since the last response from @Hubert Dudek​, and I was checking back to see if his suggestions helped you.Or else, If you have any solution, please do share that with the community as it ...
- 2 kudos