- 5790 Views
- 4 replies
- 5 kudos
Resolved! High Concurrency Pass Through Cluster : pyarrow optimization not working while converting to pandasdf
i need to convert a spark dataframe to pandas dataframe with arrow optimization spark.conf.set("spark.sql.execution.arrow.enabled", "true")data_df=df.toPandas()but getting one of the below error randomly while doing so Exception: arrow is not support...
- 5790 Views
- 4 replies
- 5 kudos
Latest Reply
- 5 kudos
Can you confirm this is a known issue?Running into same issue, example to test in 1 cell.# using Arrow fails on HighConcurrency-cluster with PassThrough in runtime 10.4 (and 10.5 and 11.0) spark.conf.set("spark.sql.execution.arrow.pyspark.enabled",...
- 5 kudos