cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

Overwrite Job Errors

inaafina
Visitor

I keep on getting this error message when trying to create a parquet in my notebook. Has anyone found this error too? My overwrite path was working perfectly fine before.

SparkConnectGrpcException: (java.lang.IllegalStateException) operationId: d5871ddd-4b1c-4838-a074-9a42a66e28ee with status ReadyForExecution is not within statuses List(Finished, Failed, Canceled) for event Closed

 

1 REPLY 1

Satyasai
Contributor

@inaafina 

1. Upgrade Databricks Connect and gRPC Packages
If you are using an external IDE to run this workflow using Databricks Connect then you may experience gRPC state race conditions if your client is out of date. Upgrade your local Python packages to the latest versions.

Bash
pip install --upgrade databricks-connect grpcio grpcio-status
Databricks have made several improvements in their clients to improve stability around proxies dropping and sessions throwing exceptions.

2. Optimize or Repartition Large Overwrites
If you are overwriting a large dataset then you may be reaching the timeout or payload limits on the gRPC channel due to the high volume of data being serialised.

You can re-partition your data before overwriting. This will reduce any skew in your partitions or split large datasets into smaller ones.

Python
df.repartition(200).write.mode("overwrite").parquet("path/to/data")

3. Increase Timeout for Spark Connect Client
If you are taking a long time to stage your writes then you may get a timeout on the gRPC channel. You can increase these timeout values using your Spark session configurations.

Python
spark.conf.set("spark.connect.client.session.timeout", "600s")