I am writing/reading data from Azure databricks to data lake. I wrote dataframe to a path in delta format using query a below, later I realized that I need the data in parquet format, and I went to the storage account and manually deleted the filepath. Now, when I try to execute query b it always throwing an error c below. I am pretty sure the filepath now does not exists on the storage because I manually deleted it. What is missing here, is this some kind of bug? Thanks in advance!
a) df.coalesce(1).write.format('delta').mode('overwrite').option('overwriteSchema', 'true').save(filepath)
b) df.coalesce(1).write.format('parquet').mode('overwrite').option('overwriteSchema', 'true').save(filepath)
c) AnalysisException: Incompatible format detected.
A transaction log for Databricks Delta was found at `filepath_delta_log`,
but you are trying to write to `filepath` using format("parquet"). You must use
'format("delta")' when reading and writing to a delta table.
