Data Engineering

by Maksym • New Contributor III

01-19-2022 1:36:14 AM

6320 Views
4 replies
7 kudos

Resolved! Databricks Autoloader is getting stuck and does not pass to the next batch

I have a simple job scheduled every 5 min. Basically it listens to cloudfiles on storage account and writes them into delta table, extremely simple. The code is something like this:df = (spark .readStream .format("cloudFiles") .option('cloudFil...

Data Engineering

6320 Views
4 replies
7 kudos

01-19-2022 1:36:14 AM

View Replies

Latest Reply

lassebe
New Contributor II

08-31-2023 1:22:00 AM

7 kudos

I had the same issue: files would randomly not be loaded.Setting `.option("cloudFiles.useIncrementalListing", False)` Seemed to do the trick!

7 kudos

08-31-2023 1:22:00 AM

3 More Replies

by Olli • New Contributor III

01-11-2022 3:17:25 AM

2371 Views
3 replies
0 kudos

Resolved! Autoloader streams fail unable to locate checkpoint/metadata or metadata/rocksdb/SSTs/sst files after interruption from cluster termination

I have a pipeline with + 20 streams running based on autoloader. The pipeline crashed and after the crash I'm unable to start the streams and they fail with one of the following messages:1): The metadata file in the streaming source checkpoint direct...

Data Engineering

2371 Views
3 replies
0 kudos

01-11-2022 3:17:25 AM

View Replies

Latest Reply

Anonymous
Not applicable

01-17-2022 9:29:56 AM

0 kudos

@Olli Tiihonen - Thanks for letting us know. I'm glad you were able to get to the bottom of things.

0 kudos

01-17-2022 9:29:56 AM

2 More Replies

Databricks Community

Forum Posts

Resolved! Databricks Autoloader is getting stuck and does not pass to the next batch

Resolved! Autoloader streams fail unable to locate checkpoint/metadata or metadata/rocksdb/SSTs/sst files after interruption from cluster termination