cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

MRTN
by New Contributor III
  • 1533 Views
  • 1 replies
  • 1 kudos

Columns archive_time, commit_time, archive_time always NULL when running cloud_files_state

Am attempting to find the commit_time for a given file for a delta table using the cloud_files_state command. However, the archive_time, commit_time, and archive_time coluns are always NULL. I am running databrics runtime 11.3 and have also verified ...

cloud_files_state
  • 1533 Views
  • 1 replies
  • 1 kudos
Latest Reply
Anonymous
Not applicable
  • 1 kudos

@Morten Stakkeland​ :The issue you are facing with the cloud_files_state command is a known limitation in Delta Lake as of the latest stable release (Delta Lake 1.0). The commit_time and protocol columns are always null, and the archive_time column i...

  • 1 kudos
Benji0934
by New Contributor II
  • 2174 Views
  • 2 replies
  • 3 kudos

Auto Loader: Empty fields (discovery_time, commit_time, archive_time) in cloud_files_state

Hi! Why are the fields discovery_time, commit_time, and archive_time NULL in cloud_files_state? Do I need to configure anything when creating my Auto Loader? df = spark.readStream.format("cloudFiles") \ .option("cloudFiles.format", "json") \ ...

  • 2174 Views
  • 2 replies
  • 3 kudos
Latest Reply
Hubert-Dudek
Esteemed Contributor III
  • 3 kudos

Please be sure that the DBR version is 10.5 or highercommit_time and archive_time can be null but discovery_time is set even as NOT NULL in the table definition so it is a bit strange. Please change the DBR version first.

  • 3 kudos
1 More Replies
Labels