Autoloader: Read old version of file. Read modification time is X, latest modification time is X

[stevenayers-bge]

I'm recieving this error from autoloader. It seems to be stuck on this one file. I don't care when it was read and last modified, I just want to ingest it. Any ideas?

java.io.IOException: Read old version of file s3a://<file-path>.json. Read modification time is 1713910814000, latest modification time is 1713925112000

at com.databricks.sql.io.StalenessChecker$Impl.check(StalenessChecker.java:223) at com.databricks.photon.NativeIOBroker.lambda$new$0(NativeIOBroker.java:374)

at org.apache.spark.TaskContextImpl.$anonfun$invokeTaskCompletionListeners$1(TaskContextImpl.scala:173)

at org.apache.spark.TaskContextImpl.$anonfun$invokeTaskCompletionListeners$1$adapted(TaskContextImpl.scala:173)

at org.apache.spark.TaskContextImpl.invokeListeners(TaskContextImpl.scala:228)

[PotnuruSiva]

@stevenayers-bge Autoloader is designed to work best with immutable files. If files are mutable (i.e., they can be updated), it is recommended to set cloudFiles.allowOverwrites = true to ensure that the latest version of the file is read.

Please refer to the below documentation for more details:

https://docs.databricks.com/en/ingestion/cloud-object-storage/auto-loader/faq.html#does-auto-loader-...