cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Olli
by New Contributor III
  • 3129 Views
  • 3 replies
  • 0 kudos

Resolved! Autoloader streams fail unable to locate checkpoint/metadata or metadata/rocksdb/SSTs/sst files after interruption from cluster termination

I have a pipeline with + 20 streams running based on autoloader. The pipeline crashed and after the crash I'm unable to start the streams and they fail with one of the following messages:1): The metadata file in the streaming source checkpoint direct...

  • 3129 Views
  • 3 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

@Olli Tiihonen​  - Thanks for letting us know. I'm glad you were able to get to the bottom of things.

  • 0 kudos
2 More Replies
dimoobraznii
by New Contributor III
  • 6407 Views
  • 5 replies
  • 6 kudos

Resolved! Autoloader failed

I used autoloader with TriggerOnce = true and ran it for weeks with schedule. Today it broke:The metadata file in the streaming source checkpoint directory is missing. This metadatafile contains important default options for the stream, so the stream...

  • 6407 Views
  • 5 replies
  • 6 kudos
Latest Reply
Deepak_Bhutada
Contributor III
  • 6 kudos

Hi dimoobraznii (Customer)This error comes in streaming when someone makes changes to the streaming checkpoint directory manually or points some streaming type to the checkpoint of some other streaming type. Please check if any changes were made to t...

  • 6 kudos
4 More Replies
Labels