I am exporting parquet files (partitioned by id) in append mode. However, I encounter errors occasionally, while other times the job completes successfully.

Apache Spark Exception: Exception thrown in awaitResult: hadoop_azure_shaded.com.microsoft.azure.storage.StorageException: The specified blob does not exist.

Currently, the storage access is configured as follows:`wasbs://<container-name>@<storage-account-name>.blob.core.windows.net/<directory-name>`

For exporting using append mode. anyone can help?