topic Re: ETL Pipeline work fine, but when executed via Workflow it fails due to StorageAccessError in Data Engineering

ETL Pipeline work fine, but when executed via Workflow it fails due to StorageAccessError

xx123 — Thu, 06 Mar 2025 18:27:21 GMT

I have a fairly simple ETL Pipeline that uses dlt. It streams data from ADLS2 SA and creates materialized view using two tables. It works fine when i execute it on its own. Materialized view is properly refreshed.

Now I wanted to add this as a task to Workflow and I did so, but when i execute the workflow, it fails with the error:

Run result unavailable: run failed with error message 
The Azure container does not exist

Any ideas how to troubleshoot/solve it?

Thanks!

Re: ETL Pipeline work fine, but when executed via Workflow it fails due to StorageAccessError

Nivethan_Venkat — Sun, 09 Mar 2025 15:34:57 GMT

Hi @xx123,

Could you please provide more details on the cluster configuration?. I am guessing the cluster policy you might be using for deploying the job in workflow / and when you are testing might be different. Please try to use same cluster policy when deploying the pipeline via workflow. Hopefully, this might resolve the issue, please let us know if this resolves after your testing.

Best Regards,
Nivethan V