Hi,

Is it possible that you tried to create that table before and that failed? Or even that there was a failure this time in creating the table? Our open source team seems this problem sometimes, and the error message is misleading. Basically, there may be a run when you try to create the table, but that fails. The file created by the failed task gets uploaded to S3, and then any retries will see that file and report that the file already exists. I suggest two best practices for preventing this:

1) Make sure you get rid of possible corrupt files.

a) Always blindly delete the table directory when you want to overwrite it in case there are leftover corrupt files.

b) Wrap your table creation in a try-catch block. If it fails, catch the exception and clean up the folder.

2) When you do get this table write error - you should go to the Spark cluster UI and drill down to the task that failed to understand the real error. Just relying on the error message in the notebook is not enough.