Autoloader: Read old version of file. Read modification time is X, latest modification time is X
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
04-26-2024 01:28 AM
I'm recieving this error from autoloader. It seems to be stuck on this one file. I don't care when it was read and last modified, I just want to ingest it. Any ideas?
java.io.IOException: Read old version of file s3a://<file-path>.json. Read modification time is 1713910814000, latest modification time is 1713925112000
at com.databricks.sql.io.StalenessChecker$Impl.check(StalenessChecker.java:223) at com.databricks.photon.NativeIOBroker.lambda$new$0(NativeIOBroker.java:374)
at org.apache.spark.TaskContextImpl.$anonfun$invokeTaskCompletionListeners$1(TaskContextImpl.scala:173)
at org.apache.spark.TaskContextImpl.$anonfun$invokeTaskCompletionListeners$1$adapted(TaskContextImpl.scala:173)
at org.apache.spark.TaskContextImpl.invokeListeners(TaskContextImpl.scala:228)
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-04-2024 06:47 AM
@stevenayers-bge Autoloader is designed to work best with immutable files. If files are mutable (i.e., they can be updated), it is recommended to set cloudFiles.allowOverwrites = true to ensure that the latest version of the file is read.
Please refer to the below documentation for more details:
