- 3215 Views
- 2 replies
- 2 kudos
Resolved! How do I use numpy case when condition in pyspark.pandas?
I do have some legacy pandas codes which I want to migrate to spark to leaverage parellelization in Databricks. I see datadricks has launched a wrapper package on top of pandas which uses pandas nomenclature but use spark engine in the backend.I comf...
- 3215 Views
- 2 replies
- 2 kudos
- 2 kudos
Hi @mahesh vardhan gandhi​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from ...
- 2 kudos
- 13793 Views
- 4 replies
- 3 kudos
[Pyspark.Pandas] PicklingError: Could not serialize object (this error is happening only for large datasets)
Context: I am using pyspark.pandas in a Databricks jupyter notebook and doing some text manipulation within the dataframe..pyspark.pandas is the Pandas API on Spark and can be used exactly the same as usual PandasError: PicklingError: Could not seria...
- 13793 Views
- 4 replies
- 3 kudos
- 3 kudos
@Krishna Zanwar​ , i'm receiving the same error.​For me, the behavior is when trying to broadcast a random forest (sklearn 1.2.0) recently loaded from mlflow, and using Pandas UDF to predict a model.​However, the same code works perfectly on Spark 2....
- 3 kudos