Databricks job runs failures Py4JJavaError: An error occurred while calling o404.sql. : java.util.No

[vishalv4476]

Hi ,

We had a successful running pipeline but it started failing since 20th august , no change were published. Can you please guide me resolve this issue.

I've tried increasing delta.deletedFileRetentionDuration' = 'interval 365 days' but it didn't helped.

Error logs: 

Py4JJavaError: An error occurred while calling o404.sql. : java.util.NoSuchElementException: None.get at scala.None$.get(Option.scala:529) at scala.None$.get(Option.scala:527) at com.databricks.sql.transaction.tahoe.commands.VacuumCommand$.$anonfun$gc$16(VacuumCommand.scala:569) at com.databricks.sql.util.ThreadLocalTagger.withTag(QueryTagger.scala:62) at com.databricks.sql.util.ThreadLocalTagger.withTag$(QueryTagger.scala:59) at com.databricks.sql.util.QueryTagger$.withTag(QueryTagger.scala:159) at com.databricks.sql.transaction.tahoe.metering.DeltaLogging.$anonfun$recordFrameProfileWithDmqTag$1(DeltaLogging.scala:382) at com.databricks.spark.util.FrameProfiler$.record(FrameProfiler.scala:94) at com.databricks.sql.transaction.tahoe.metering.DeltaLogging.recordFrameProfileWithDmqTag(DeltaLogging.scala:382) at com.databricks.sql.transaction.tahoe.metering.DeltaLogging.recordFrameProfileWithDmqTag$(DeltaLogging.scala:381) at com.databricks.sql.transaction.tahoe.commands.VacuumCommand$.recordFrameProfileWithDmqTag(VacuumCommand.scala:70) at com.databricks.sql.transaction.tahoe.commands.VacuumCommand$.$anonfun$gc$2(VacuumCommand.scala:351) at com.databricks.sql.transaction.tahoe.metering.DeltaLogging.withOperationTypeTag(DeltaLogging.scala:325) at com.databricks.sql.transaction.tahoe.metering.DeltaLogging.withOperationTypeTag$(DeltaLogging.scala:312) at com.databricks.sql.transaction.tahoe.commands.VacuumCommand$.withOperationTypeTag(VacuumCommand.scala:70) at com.databricks.sql.transaction.tahoe.metering.DeltaLogging.$anonfun$recordDeltaOperationInternal$2(DeltaLogging.scala:178) at com.databricks.spark.util.FrameProfiler$.record(FrameProfiler.scala:94) at com.databricks.sql.transaction.tahoe.metering.DeltaLogging.recordFrameProfile(DeltaLogging.scala:418) at com.databricks.sql.transaction.tahoe.metering.DeltaLogging.recordFrameProfile$(DeltaLogging.scala:416) at com.databricks.sql.transaction.tahoe.commands.VacuumCommand$.recordFrameProfile(VacuumCommand.scala:70) at com.databricks.sql.transaction.tahoe.metering.DeltaLogging.$anonfun$recordDeltaOperationInternal$1(DeltaLogging.scala:177) at com.databricks.logging.UsageLogging.$anonfun$recordOperation$1(UsageLogging.scala:528) at com.databricks.logging.UsageLogging.executeThunkAndCaptureResultTags$1(UsageLogging.scala:633) at com.databricks.logging.UsageLogging.$anonfun$recordOperationWithResultTags$4(UsageLogging.scala:656) at com.databricks.logging.AttributionContextTracing.$anonfun$withAttributionContext$1(AttributionContextTracing.scala:48)

[SP_6721]

Hi @vishalv4476 ,

The error is likely due to a corrupted Delta transaction log or files deleted manually/outside of Delta. Check the table history and verify that no user or automated process removed data files. If issues are found, restore the table to a previous working version, also changing retention settings will not recover already deleted files.