- 6949 Views
- 4 replies
- 7 kudos
Resolved! Databricks Autoloader is getting stuck and does not pass to the next batch
I have a simple job scheduled every 5 min. Basically it listens to cloudfiles on storage account and writes them into delta table, extremely simple. The code is something like this:df = (spark .readStream .format("cloudFiles") .option('cloudFil...
- 6949 Views
- 4 replies
- 7 kudos
- 7 kudos
I had the same issue: files would randomly not be loaded.Setting `.option("cloudFiles.useIncrementalListing", False)` Seemed to do the trick!
- 7 kudos
- 2677 Views
- 3 replies
- 0 kudos
Resolved! Autoloader streams fail unable to locate checkpoint/metadata or metadata/rocksdb/SSTs/sst files after interruption from cluster termination
I have a pipeline with + 20 streams running based on autoloader. The pipeline crashed and after the crash I'm unable to start the streams and they fail with one of the following messages:1): The metadata file in the streaming source checkpoint direct...
- 2677 Views
- 3 replies
- 0 kudos
- 0 kudos
@Olli Tiihonen​ - Thanks for letting us know. I'm glad you were able to get to the bottom of things.
- 0 kudos