Autoloader streams fail unable to locate checkpoint/metadata or metadata/rocksdb/SSTs/sst files after interruption from cluster termination

[Olli]

I have a pipeline with + 20 streams running based on autoloader. The pipeline crashed and after the crash I'm unable to start the streams and they fail with one of the following messages:

1):

The metadata file in the streaming source checkpoint directory is missing. This metadata file contains important default options for the stream, so the stream cannot be restarted right now. Please contact Databricks support for assistance.

2):

Caused by: java.io.FileNotFoundException: No such file or directory: s3a://elisa-automate-ml/pipeline-kpis/kpi_polystar_map/checkpoint/sources/0/rocksdb/SSTs/000061-fc068971-99e3-403e-8767-2335ec7a0330.sst

[Anonymous]

Hello @Olli Tiihonen​! Welcome and thanks for your question. We'll make sure the community has a chance to answer your question before we come back around. Thanks!

[Olli · Accepted solution]

seems like the root problem was having checkpoint located in the data folder and running vacuum on the delta table. Had to restart streams with fresh checkpoint locations to get this sorted.

[Anonymous]

@Olli Tiihonen​  - Thanks for letting us know. I'm glad you were able to get to the bottom of things. 🙂