topic Re: DeltaFileNotFoundException: No file found in the directory (sudden task failure) in Data Engineering

DeltaFileNotFoundException: No file found in the directory (sudden task failure)

Juju — Mon, 20 Nov 2023 02:29:33 GMT

Hi all,

I am currently running a job that will upsert a table by reading from delta change data feed from my silver table. Here is the relevent snippet of code:

rds_changes = spark.read.format("delta") \ .option("readChangeFeed", "true") \ .option("startingVersion", 0) \ .table("main.default.gold_table") \ .where(f"_commit_timestamp >= '{(datetime.now() - timedelta(hours=1)).strftime('%Y-%m-%d %H:%M:%S')}'")

Here is the error returned

com.databricks.sql.transaction.tahoe.DeltaFileNotFoundException: No file found in the directory: s3://databricks-workspace-stack-70da1-metastore-bucket/60ed403c-0a54-4f42-8b8a-73b8cea1bdc3/tables/6d4a9b3d-f88b-436e-be1b-09852f605f4c/_delta_log.

I have done the following:

Verify that the delta log folder is not deleted by accessing S3 directly
Able to query the table directly and perform `DESCRIBE HISTORY gold_table` on it without any issue

Anyone has any idea why this happen when I am running the job which was working fine previously without any changes

Re: DeltaFileNotFoundException: No file found in the directory (sudden task failure)

Juju — Mon, 20 Nov 2023 08:59:52 GMT

Hey @Retired_mod , found the issue is due to truncated delta log. Thanks for the help man

Re: DeltaFileNotFoundException: No file found in the directory (sudden task failure)

c-data — Tue, 22 Oct 2024 20:39:52 GMT

What was the fix?

Re: DeltaFileNotFoundException: No file found in the directory (sudden task failure)

Walecon — Thu, 14 Nov 2024 09:42:02 GMT

1) check the first delta feed enabled version in
DESCRIBE HISTORY `table_name`;

2) use this version instead of 0 in .option("startingVersion", x)

Re: DeltaFileNotFoundException: No file found in the directory (sudden task failure)

sanpai — Thu, 06 Feb 2025 14:19:57 GMT

Can you please describe some more on option 2)

Re: DeltaFileNotFoundException: No file found in the directory (sudden task failure)

Walecon — Thu, 06 Feb 2025 15:41:22 GMT

You would run DESCRIBE HISTORY `table_name`; to check which versions are available. If the delta log is truncated for some reason, you will not find a version 0. Use the oldest version you can find instead of 0. For example, if the oldest version you can find in the delta log is 10, use .option("startingVersion", 10).