cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Failed to read job commit marker error

ak4
New Contributor II

Recently, we migrate from DBR 11.3 LTS ML to DBR 14.3 LTS ML. We are struggling on one data source where we consume parquet files. New data are appended every 30 minutes to that data source. The data are generated by Databricks notebook which runs on even lower runtime than we used for reading before (I don't have exact information, but probably DBR 10.4). Before (with DBR 11.3), there was no issue, but with new runtime (14.3), we encounter randomly (sometimes it works, sometimes it doesn't) this error:

java.io.IOException: Failed to read job commit marker: FileStatus{path=dbfs:/mnt/prod_profiles/parsed/personal/_committed_8360981374007134857; isDirectory=false; length=122; replication=1; blocksize=536870912; modification_time=1725455799000; access_time=0; owner=; group=; permission=rwx-wx-wx; isSymlink=false; hasAcl=false; isEncrypted=false; isErasureCoded=false}

From the observation so far, I have a theory that the issue happens when we try to read the data and new data are appended to that source at the same time. Can anyone help and suggest what we can do? Unfortunately, we can't affect which runtime is used for that data source, we can affect only our runtime for reading the data (which we would like not to downgrade back to 11.3) and our code.

2 REPLIES 2

menotron
Valued Contributor

Hi @ak4, this is an expected behavior with DBIO transactional commit enabled and most likely the issue is when you update a table and query it immediately. You could explicitly invalidate the cache or configure disk cache. 

If you have a long running job and the underlying files are updated by a separate job you can invalidate the cache by running 'REFRESH TABLE tableName' command in SQL or by recreating the Dataset/DataFrame involved. If Delta cache is stale or the underlying files have been removed, you can invalidate Delta cache manually by restarting the cluster.

ak4
New Contributor II

Thanks @menotron from your reply!

Interestingly, we have been using REFRESH TABLE command even before this issue and it worked well so far. However, now with new runtime, it doesn't work anymore. I should specify the code which we use. It actually fails in different versions of our code:

1) refresh+reading hive table

spark.sql('refresh table ' + table_name)
df = spark.table(table_name)
 
2) reading directly from storage
df = spark.read.parquet("dbfs:/mnt/prod_profiles/parsed/personal")
 
3) reading data as streaming (it is an old code when autoloader wasn't in place) - the error occurs when trying to write data into delta
df = spark.readStream.format('parquet').schema(schema).option('path', input_path).load()
(df.writeStream
.format("delta")
.option("checkpointLocation", checkpoint_path)
.option("path", target_path)
.option("mode", "append")
.trigger(once=True)
.start())
 
About your last suggestion, I am not sure if I understand exactly what you mean:
If Delta cache is stale or the underlying files have been removed, you can invalidate Delta cache manually by restarting the cluster.
We have this code as part of our pipeline, so we are searching to some automatic solution. For now, we are trying to use simple retry but I don't think it is ideal because it is not ensuring 100 % success and makes our pipeline potentially longer. Another thing we tried (but with not high hopes that it can actually help), was even higher runtime 15.4 LTS ML. The result was the same...
 
Any other suggestions highly appreciated if more details helped to explain the issue better.

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group