Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
Showing results for 
Search instead for 
Did you mean: 

Auto Loader: Empty fields (discovery_time, commit_time, archive_time) in cloud_files_state

New Contributor II


Why are the fields discovery_time, commit_time, and archive_time NULL in cloud_files_state?

Do I need to configure anything when creating my Auto Loader?

df = spark.readStream.format("cloudFiles") \
    .option("cloudFiles.format", "json") \
    .option("cloudFiles.tenantId", tenantId) \
    .option("cloudFiles.clientId", clientId) \
    .option("cloudFiles.clientSecret", clientSecret) \
    .option("cloudFiles.resourceGroup", resourceGroup) \
    .option("cloudFiles.subscriptionId", subscriptionId) \
    .option("cloudFiles.useNotifications", "true") \
    .option("cloudFiles.includeExistingFiles", "true") \
    .option("cloudFiles.schemaLocation", checkpoint_path) \
    .option("cloudFiles.schemaEvolutionMode", "rescue") \
    .option("recursiveFileLookup", "true") \
    .option("badRecordsPath", bad_records_path)
    .option("multiLine", "true")
    .schema(dfSchema.schema) \
#Transforming dataframe stream...
df6.writeStream \
    .format("delta") \
    .foreachBatch(upsertToDelta) \
    .option("checkpointLocation", checkpoint_path) \
    .outputMode("update") \
    .start(targetDir) #target folder


Esteemed Contributor III
  • Please be sure that the DBR version is 10.5 or higher
  • commit_time and archive_time can be null but discovery_time is set even as NOT NULL in the table definition so it is a bit strange. Please change the DBR version first.

Hi Hubert,

Thank you for your reply 🙂

The DBR version is 11.3. And yes it is indeed very strange.


Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!