Testing Databricks Auto Loader File Notification (File Event) in Public Preview - Spark Termination

[dawn-dot-py]

I tried to test the Databricks Auto Loader file notification (file event) feature, which is currently in public preview, using a notebook for work purposes. However, when I ran display(df), Spark terminated and threw the error shown in the attached image.

Is the file event mode in the public preview phase currently not operational? I am still learning about Databricks, so I am asking here for help.

[Advika · Accepted solution]

Hello @dawn-dot-py!

Auto Loader’s managed file events are indeed in Public Preview, but they’re available to allowlisted workspaces. The error you encountered means your workspace hasn’t been enrolled in the preview, which is expected unless you’ve been granted access.
To use this feature, you can reach out to Databricks Support or your account team to request enrollment. Until then, you can set cloudFiles.useManagedFileEvents=false.

https://docs.databricks.com/aws/en/ingestion/cloud-object-storage/auto-loader/file-notification-mode...