Addressing Pipeline Error Handling in Databricks bundle run with CI/CD when SUCCESS WITH FAILURES

ismaelhenzel — Mon, 08 Apr 2024 14:28:48 GMT

I'm using Databricks asset bundles and I have pipelines that contain "if all done rules". When running on CI/CD, if a task fails, the pipeline returns a message like "the job xxxx SUCCESS_WITH_FAILURES" and it passes, potentially deploying a broken pipe in production. I would prefer that the CI/CD throws an error in these cases, not mark it as success. Is there a way to do this, like a parameter in a bundle run? If not, should I recreate my pipe in production with "all done" rules, but in development with "all succeeded" to capture the errors in CI/CD? I understand that I should have a QA environment to test these cases, but unfortunately, that's not the case right now.

Re: Addressing Pipeline Error Handling in Databricks bundle run with CI/CD when SUCCESS WITH FAILURE

ismaelhenzel — Tue, 23 Apr 2024 11:28:36 GMT

Awesome answer, I will try the first approach. I think it is a less intrusive solution than changing the rules of my pipeline in development scenarios. This way, I can maintain a general pipeline for deployment across all environments. We plan to implement a QA environment after migrating all cloud resources to Terraform. Thanks!

topic Re: Addressing Pipeline Error Handling in Databricks bundle run with CI/CD when SUCCESS WITH FAILURE in Data Engineering

Addressing Pipeline Error Handling in Databricks bundle run with CI/CD when SUCCESS WITH FAILURES

Re: Addressing Pipeline Error Handling in Databricks bundle run with CI/CD when SUCCESS WITH FAILURE