Testing Databricks Auto Loader File Notification (File Event) in Public Preview - Spark Termination

dawn-dot-py — Thu, 07 Aug 2025 04:58:06 GMT

I tried to test the Databricks Auto Loader file notification (file event) feature, which is currently in public preview, using a notebook for work purposes. However, when I ran display(df), Spark terminated and threw the error shown in the attached image.

Is the file event mode in the public preview phase currently not operational? I am still learning about Databricks, so I am asking here for help.

Re: Testing Databricks Auto Loader File Notification (File Event) in Public Preview - Spark Terminat

Advika — Fri, 08 Aug 2025 11:04:33 GMT

Hello @dawn-dot-py!

Auto Loader’s managed file events are indeed in Public Preview, but they’re available to allowlisted workspaces. The error you encountered means your workspace hasn’t been enrolled in the preview, which is expected unless you’ve been granted access.
To use this feature, you can reach out to Databricks Support or your account team to request enrollment. Until then, you can set cloudFiles.useManagedFileEvents=false.

https://docs.databricks.com/aws/en/ingestion/cloud-object-storage/auto-loader/file-notification-mode#-use-file-notification-mode-with-file-events

topic Testing Databricks Auto Loader File Notification (File Event) in Public Preview - Spark Termination in Data Engineering

Testing Databricks Auto Loader File Notification (File Event) in Public Preview - Spark Termination

Re: Testing Databricks Auto Loader File Notification (File Event) in Public Preview - Spark Terminat