- 15773 Views
- 3 replies
- 3 kudos
[Pyspark.Pandas] PicklingError: Could not serialize object (this error is happening only for large datasets)
Context: I am using pyspark.pandas in a Databricks jupyter notebook and doing some text manipulation within the dataframe..pyspark.pandas is the Pandas API on Spark and can be used exactly the same as usual PandasError: PicklingError: Could not seria...
- 15773 Views
- 3 replies
- 3 kudos
- 3 kudos
@Krishna Zanwar​ , i'm receiving the same error.​For me, the behavior is when trying to broadcast a random forest (sklearn 1.2.0) recently loaded from mlflow, and using Pandas UDF to predict a model.​However, the same code works perfectly on Spark 2....
- 3 kudos
- 1425 Views
- 1 replies
- 2 kudos
When to use Dataframes API over Spark SQL
Hello Experts,I am new to Databricks. Building data pipelines, I have both batch and streaming data.Should I use Dataframes API to read csv files then convert to parquet format then do the transformation? orwrite to table using CSV then use Spark SQL...
- 1425 Views
- 1 replies
- 2 kudos
- 2 kudos
Hi Rathinam, It would be better to understand the pipeline more in this situation. Writing to table using CSV and then using spark SQL will be faster in few cases than the other one.
- 2 kudos