Autoloader: Read old version of file. Read modification time is X, latest modification time is X

stevenayers-bge — Fri, 26 Apr 2024 08:28:05 GMT

I'm recieving this error from autoloader. It seems to be stuck on this one file. I don't care when it was read and last modified, I just want to ingest it. Any ideas?

java.io.IOException: Read old version of file s3a://<file-path>.json. Read modification time is 1713910814000, latest modification time is 1713925112000

at com.databricks.sql.io.StalenessChecker$Impl.check(StalenessChecker.java:223) at com.databricks.photon.NativeIOBroker.lambda$new$0(NativeIOBroker.java:374)

at org.apache.spark.TaskContextImpl.$anonfun$invokeTaskCompletionListeners$1(TaskContextImpl.scala:173)

at org.apache.spark.TaskContextImpl.$anonfun$invokeTaskCompletionListeners$1$adapted(TaskContextImpl.scala:173)

at org.apache.spark.TaskContextImpl.invokeListeners(TaskContextImpl.scala:228)

Re: Autoloader: Read old version of file. Read modification time is X, latest modification time is X

PotnuruSiva — Wed, 04 Dec 2024 14:47:42 GMT

@stevenayers-bge Autoloader is designed to work best with immutable files. If files are mutable (i.e., they can be updated), it is recommended to set cloudFiles.allowOverwrites = true to ensure that the latest version of the file is read.

Please refer to the below documentation for more details:

https://docs.databricks.com/en/ingestion/cloud-object-storage/auto-loader/faq.html#does-auto-loader-process-the-file-again-when-the-file-gets-appended-or-overwritten

topic Autoloader: Read old version of file. Read modification time is X, latest modification time is X in Data Engineering

Autoloader: Read old version of file. Read modification time is X, latest modification time is X

Re: Autoloader: Read old version of file. Read modification time is X, latest modification time is X