https://community.databricks.com/t5/get-started-discussions/getting-filenotfoundexception-while-using-cloudfiles/m-p/43638#M932 <P>Danny is another process mutating / deleting the incoming files?</P><P>&nbsp;</P> Tue, 05 Sep 2023 13:05:02 GMT BilalAslamDbrx 2023-09-05T13:05:02Z https://community.databricks.com/t5/get-started-discussions/getting-filenotfoundexception-while-using-cloudfiles/m-p/43626#M931 <P>Hi,<BR />Following is the code i am using the ingest the data incrementally (weekly).</P><DIV><DIV><SPAN>val</SPAN> <SPAN>ssdf</SPAN><SPAN> = spark.readStream</SPAN><SPAN>.schema(</SPAN><SPAN>schema</SPAN><SPAN>)&nbsp;</SPAN></DIV><DIV><SPAN>.format(</SPAN><SPAN>"cloudFiles"</SPAN><SPAN>)</SPAN></DIV><DIV><SPAN>.option(</SPAN><SPAN>"cloudFiles.format"</SPAN><SPAN>, </SPAN><SPAN>"parquet"</SPAN><SPAN>)</SPAN></DIV><DIV><SPAN>.load(sourceUrl)</SPAN></DIV><DIV><SPAN>.filter(criteriaFilter</SPAN><SPAN><SPAN>)<BR /><BR />val transformedDf = ssdf.transform(.....)<BR /><BR /></SPAN></SPAN><DIV><DIV><SPAN>val</SPAN> <SPAN>processData</SPAN><SPAN> = transformedDf</SPAN></DIV><DIV><SPAN>.select(</SPAN><SPAN>recordFields</SPAN><SPAN>: </SPAN><SPAN>_*</SPAN><SPAN>)&nbsp;</SPAN></DIV><DIV><SPAN>.writeStream</SPAN></DIV><DIV><SPAN>.option(</SPAN><SPAN>"checkpointLocation"</SPAN><SPAN>, outputUrl + </SPAN><SPAN>"checkpoint/"</SPAN><SPAN>)</SPAN></DIV><DIV><SPAN>.format(</SPAN><SPAN>"parquet"</SPAN><SPAN>)</SPAN></DIV><DIV><SPAN>.outputMode(</SPAN><SPAN>"append"</SPAN><SPAN>)</SPAN></DIV><DIV><SPAN>.option(</SPAN><SPAN>"path"</SPAN><SPAN>, outputUrl + run_id + </SPAN><SPAN>"/"</SPAN><SPAN>)&nbsp;</SPAN></DIV><DIV><SPAN>.trigger(</SPAN><SPAN>Trigger</SPAN><SPAN>.</SPAN><SPAN>Once</SPAN><SPAN>())</SPAN></DIV><DIV><SPAN>.start()</SPAN></DIV><DIV><SPAN>processData.processAllAvailable()</SPAN></DIV><DIV><SPAN>processData.stop()<BR /><BR />For each week, the data is written to a new folder and checkpoint to the same folder.<BR />This worked fine for 3 to 5 incremental run.<BR />But recently i got the following error :&nbsp;&nbsp;<BR />ERROR: Query termination received for [id=2345245425], with exception: org.apache.spark.SparkException: Job aborted.<BR />Caused by: java.io.FileNotFoundException: Unable to find batch s3://outputPath/20230810063959/_spark_metadata/0<BR />What is the reason for this issue ? Any idea?</SPAN></DIV></DIV></DIV></DIV> Tue, 05 Sep 2023 11:14:42 GMT https://community.databricks.com/t5/get-started-discussions/getting-filenotfoundexception-while-using-cloudfiles/m-p/43626#M931 dannythermadom 2023-09-05T11:14:42Z https://community.databricks.com/t5/get-started-discussions/getting-filenotfoundexception-while-using-cloudfiles/m-p/43638#M932 <P>Danny is another process mutating / deleting the incoming files?</P><P>&nbsp;</P> Tue, 05 Sep 2023 13:05:02 GMT https://community.databricks.com/t5/get-started-discussions/getting-filenotfoundexception-while-using-cloudfiles/m-p/43638#M932 BilalAslamDbrx 2023-09-05T13:05:02Z https://community.databricks.com/t5/get-started-discussions/getting-filenotfoundexception-while-using-cloudfiles/m-p/43640#M933 <P>New files gets added to the input location. Input files are not deleted or updated ..&nbsp;</P> Tue, 05 Sep 2023 13:08:46 GMT https://community.databricks.com/t5/get-started-discussions/getting-filenotfoundexception-while-using-cloudfiles/m-p/43640#M933 dannythermadom 2023-09-05T13:08:46Z