Recently, we migrate from DBR 11.3 LTS ML to DBR 14.3 LTS ML. We are struggling on one data source where we consume parquet files. New data are appended every 30 minutes to that data source. The data are generated by Databricks notebook which runs on even lower runtime than we used for reading before (I don't have exact information, but probably DBR 10.4). Before (with DBR 11.3), there was no issue, but with new runtime (14.3), we encounter randomly (sometimes it works, sometimes it doesn't) this error:
java.io.IOException: Failed to read job commit marker: FileStatus{path=dbfs:/mnt/prod_profiles/parsed/personal/_committed_8360981374007134857; isDirectory=false; length=122; replication=1; blocksize=536870912; modification_time=1725455799000; access_time=0; owner=; group=; permission=rwx-wx-wx; isSymlink=false; hasAcl=false; isEncrypted=false; isErasureCoded=false}
From the observation so far, I have a theory that the issue happens when we try to read the data and new data are appended to that source at the same time. Can anyone help and suggest what we can do? Unfortunately, we can't affect which runtime is used for that data source, we can affect only our runtime for reading the data (which we would like not to downgrade back to 11.3) and our code.