Hello,
I have an ETL pipeline in Databricks that works perfectly when I execute it manually in the notebook using an all-purpose cluster. However, when I try to schedule it using a job cluster, it fails immediately with the error message: 'Azure container does not exist.'
I attempted to run the job using the same all-purpose cluster, but encountered the same error. Even a test job with a simple print command resulted in the same error. It seems I'm unable to execute job runs from this Databricks workspace.
Does anyone have any insights on what might be causing this?
Thank you very much in advance
Hey @bantarobugs
There might be a problem with the permissions or roles assigned to the user or service principal trying to access the Azure container. Please check who/what is assigned and it role/permission:
