topic Re: Autoloader streams fail unable to locate checkpoint/metadata or metadata/rocksdb/SSTs/sst files after interruption from cluster termination in Data Engineering

Autoloader streams fail unable to locate checkpoint/metadata or metadata/rocksdb/SSTs/sst files after interruption from cluster termination

Olli — Tue, 11 Jan 2022 11:17:25 GMT

I have a pipeline with + 20 streams running based on autoloader. The pipeline crashed and after the crash I'm unable to start the streams and they fail with one of the following messages:

1):

The metadata file in the streaming source checkpoint directory is missing. This metadata file contains important default options for the stream, so the stream cannot be restarted right now. Please contact Databricks support for assistance.

2):

Caused by: java.io.FileNotFoundException: No such file or directory: s3a://elisa-automate-ml/pipeline-kpis/kpi_polystar_map/checkpoint/sources/0/rocksdb/SSTs/000061-fc068971-99e3-403e-8767-2335ec7a0330.sst

Re: Autoloader streams fail unable to locate checkpoint/metadata or metadata/rocksdb/SSTs/sst files after interruption from cluster termination

Anonymous — Wed, 12 Jan 2022 00:56:28 GMT

Hello @Olli Tiihonen! Welcome and thanks for your question. We'll make sure the community has a chance to answer your question before we come back around. Thanks!

Re: Autoloader streams fail unable to locate checkpoint/metadata or metadata/rocksdb/SSTs/sst files after interruption from cluster termination

Olli — Mon, 17 Jan 2022 06:49:20 GMT

seems like the root problem was having checkpoint located in the data folder and running vacuum on the delta table. Had to restart streams with fresh checkpoint locations to get this sorted.

Re: Autoloader streams fail unable to locate checkpoint/metadata or metadata/rocksdb/SSTs/sst files after interruption from cluster termination

Anonymous — Mon, 17 Jan 2022 17:29:56 GMT

@Olli Tiihonen - Thanks for letting us know. I'm glad you were able to get to the bottom of things. 🙂