topic Re: commit time is coming as null in autoloader in Data Engineering

commit time is coming as null in autoloader

shrutikatyal — Thu, 19 Jun 2025 07:09:40 GMT

As per the databricks new feature in autoloader that we can use archival and move feature in autoloader however I am trying to use that feature using databricks 16.4.x.scala2.12 however commit time is still coming null as its mentioned in the documentation if commit time is null this feature won't work . how can I resolve it?

I am using below autoloader configuration:

df=spark.readStream.format("cloudfiles")\

.option("cloudfiles.format","json")\

.option("cloudfiles.schemaLocation",checkpoint_path)\

.option("multiLine", "True")\

.option("cloudFiles.backfillInterval","10 minutes")\

.option("cloudFiles.inferColumnTypes", "True")\

.load(ingestDirectory)

dfOutput = (

df.writeStream

.trigger(once=True)

.format("delta")

.option("mergeSchema", "true")

.option("checkpointLocation",checkpoint_path)

.start(CuratedDirectory)

)

dfOutput.awaitTermination()

Re: commit time is coming as null in autoloader

Yogesh_Verma_ — Thu, 19 Jun 2025 17:10:32 GMT

You're right — the new **archival and move feature in Auto Loader** depends on the `_commit_timestamp` column. If that value is coming as `null`, the feature won't work, as mentioned in the documentation.

To fix this, you need to make sure you're explicitly enabling the `commitTime` metadata column using the following option in your Auto Loader configuration:

```python
.option("cloudFiles.addColumns", "commitTime")
```

This ensures that the `_commit_timestamp` field gets populated during ingestion.

Here’s the corrected version of your code:

```python
df = (
spark.readStream.format("cloudfiles")
.option("cloudFiles.format", "json")
.option("cloudFiles.schemaLocation", checkpoint_path)
.option("multiLine", "true")
.option("cloudFiles.backfillInterval", "10 minutes")
.option("cloudFiles.inferColumnTypes", "true")
.option("cloudFiles.addColumns", "commitTime") # Required to get _commit_timestamp
.load(ingestDirectory)
)
```

Once this is set, `_commit_timestamp` should be populated properly, and the archival/move feature should start working as expected.

Re: commit time is coming as null in autoloader

shrutikatyal — Sat, 21 Jun 2025 07:48:04 GMT

hi Yogesh,
I am getting below error if I am trying to add this in my autoloader configuration i.e. option ("cloudFiles.addColumns", "commitTime").

Re: commit time is coming as null in autoloader

shrutikatyal — Sat, 21 Jun 2025 07:52:04 GMT

Attached all screenshots for reference

Re: commit time is coming as null in autoloader

shrutikatyal — Sat, 21 Jun 2025 07:54:11 GMT

below i have attached screenshot for cluster configuration

Re: commit time is coming as null in autoloader

mani_22 — Sun, 22 Jun 2025 04:49:10 GMT

@shrutikatyal I believe the commit_time only functions when the cloudFiles.cleanSource option is enabled. I don't see this option present in your snippet. Could you please enable this option for the read and check?

Refer to the below documentation, which specifies that column commit_time is supported in Databricks Runtime 16.4 and above when cloudFiles.cleanSource is enabled

https://docs.databricks.com/aws/en/sql/language-manual/functions/cloud_files_state

A file might be processed but marked as committed arbitrarily later. commit_time is updated usually at the start of the next microbatch.

Re: commit time is coming as null in autoloader

shrutikatyal — Mon, 23 Jun 2025 06:25:18 GMT

Thanks, its working now

Re: commit time is coming as null in autoloader

shrutikatyal — Wed, 02 Jul 2025 12:31:24 GMT

hi,
I am interested in data bricks certified associate data Engineer certification can I get any voucher?

Re: commit time is coming as null in autoloader

TheOC — Wed, 02 Jul 2025 12:46:18 GMT

Hey @shrutikatyal
I believe the only current route to get a discount voucher would be the following:

https://community.databricks.com/t5/events/dais-2025-virtual-learning-festival-11-june-02-july-2025/ev-p/119323
I think it’s the last day of the event so you might need to be quick!
hope this helps,
TheOC

Re: commit time is coming as null in autoloader

shrutikatyal — Mon, 14 Jul 2025 03:54:50 GMT

hi,
I have done the certification using data bricks learning festival however I haven't got any voucher yet.
Thanks