I have a pandas dataframe in my Pyspark notebook. I want to save this dataframe to my S3 bucket. I'm using the following command to save it
import boto3
df_summary.to_csv(f"s3://dataconversion/data/exclude",index=False)
but I keep getting this error: ModuleNotFoundError: No module named 'botocore.compress'
I already tried to upgrade boto3 but same error. This problem seems to be with panda libraries only. I'm able to read from CSV with spark.read.format('csv') without issues
Any suggestions?