Auto Loader stream fails on RocksDB checkpoint after enabling managed file events

[data_pulse]

Hello Community.

We recently enabled the following option on several existing Auto Loader streams:

"cloudFiles.useManagedFileEvents": "true"

Most streams continued working, but one now fails while restoring its existing checkpoint, referencing a file similar to:

<checkpoint>/sources/0/rocksdb/7129.zip

Removing cloudFiles.useManagedFileEvents and rerunning does not resolve the error, the stream still attempts to restore the same RocksDB archive.

Has anyone encountered this when enabling managed file events on an existing checkpoint? Is there a supported way to repair or migrate the checkpoint without resetting it and potentially reprocessing existing files?

[ShamenParis]

Hi @data_pulse 

I had a quick look at this problem alongside the Databricks documentation and found a supported way to recover your stream without resetting your entire checkpoint!

This error usually happens when Auto Loader tries to initialize its state for Unity Catalog managed file events for the first time. If the stream is interrupted, crashes, or times out while doing the initial full directory listing, the RocksDB state archive captures a partial state and gets corrupted. Simply removing the useManagedFileEvents option afterward doesn't fix it because the checkpoint is already in a broken state.

Fortunately, there is a documented way to bypass an invalid read position and force the stream to reconcile its state:

1. Add .option("cloudFiles.listOnStart", "true") and .option("cloudFiles.validateOptions", "false") to your streaming query.

2. Restart the stream. Auto Loader will perform a full directory listing on start and explicitly bypass the invalid/corrupted continuation token.

3. After the stream completes a successful micro-batch, stop the stream.

4. Remove both of those options from your PySpark code and restart the stream normally.

This method allows Auto Loader to fix its state without dropping the historical file ingestion history stored in your checkpoint.

For more details on how these options interact, I found these documentation pages super helpful:

• Auto Loader File Notification Mode: https://docs.databricks.com/aws/en/ingestion/cloud-object-storage/auto-loader/file-notification-mode

• Manage External Locations (File Events): https://docs.databricks.com/aws/en/connect/unity-catalog/cloud-storage/manage-external-locations#fil...

Hope this helps get your stream unblocked!

[data_pulse]

@ShamenParis 

Thanks for looking into this. I have tried listOnStart/ validateOptions but unfortunately it didn't solve the issue. The error we have got (FAILED_READ_FILE.NO_HINT on the rocksdb zip) turned out to be different failure than continuation token issue that these options address, as mentioned in the reference.

Ended up going with resetting the checkpoint and processing only incremental files from a known recovery timestamp as a workaround, which got the stream unblocked.

[ShamenParis]

I am really glad to hear you got the stream unblocked.

Thank you for sharing the exact error code (FAILED_READ_FILE.NO_HINT). That makes perfect sense—if the RocksDB .zip archive itself is physically unreadable or corrupted at the storage layer, rather than just containing an invalid continuation token, the listOnStart option wouldn't be able to help because Spark cannot even parse the state store to begin with.

Resetting the checkpoint and explicitly processing only incremental files from a known recovery timestamp is absolutely the cleanest and safest workaround for a fundamentally unrecoverable state. It is a great architectural pattern to fall back on to ensure you don't duplicate your data.

Thanks for sharing your solution; closing the loop like this will definitely help the next person who runs into a NO_HINT error on their checkpoint files!