In case anyone else stumbles across this, I was able to fix my issue by setting up an instance profile with the file notification permissions and attaching the instance profile to the job cluster. It wasn't clear from the documentation that the file notification permissions can't be set up with a role and job using storage credentials. This article helped: https://medium.com/@mattwinmill88/deploying-a-databricks-aws-end-to-end-pipeline-using-terraform-921...