commit time is coming as null in autoloader

shrutikatyal · ‎06-19-2025

As per the databricks new feature in autoloader that we can use archival and move feature in autoloader however I am trying to use that feature using databricks 16.4.x.scala2.12 however commit time is still coming null as its mentioned in the documentation if commit time is null this feature won't work . how can I resolve it?

I am using below autoloader configuration:

df=spark.readStream.format("cloudfiles")\

.option("cloudfiles.format","json")\

.option("cloudfiles.schemaLocation",checkpoint_path)\

.option("multiLine", "True")\

.option("cloudFiles.backfillInterval","10 minutes")\

.option("cloudFiles.inferColumnTypes", "True")\

.load(ingestDirectory)

dfOutput = (

df.writeStream

.trigger(once=True)

.format("delta")

.option("mergeSchema", "true")

.option("checkpointLocation",checkpoint_path)

.start(CuratedDirectory)

)

dfOutput.awaitTermination()