Hi @ismaelhenzel, Handling error scenarios in your Databricks asset bundles during CI/CD workflows is crucial to ensure robustness and prevent potentially broken deployments.
Letโs explore some options:
-
Error Handling in Databricks Asset Bundles:
- By default, when a task fails in a Databricks bundle run, the pipeline returns a message like โthe job xxxx SUCCESS_WITH_FAILURES,โ which can be misleading in CI/CD scenarios.
- Unfortunately, there isnโt a direct parameter in the bundle run to force an error status when tasks fail. However, you can adopt alternative approaches to achieve your desired behavior.
-
Approaches to Consider:
- Custom Exit Codes:
- Separate Development and Production Pipelines:
- As you mentioned, consider having separate pipelines for development and production.
- In the development pipeline, use โall succeededโ rules to capture errors during CI/CD testing.
- In the production pipeline, use โall doneโ rules for normal execution.
- This approach ensures that errors are caught during development but not in production.
-
QA Environment (Future Consideration):
- While itโs not feasible right now, having a dedicated QA environment is essential for thorough testing.
- In the long term, aim to set up a QA environment where you can validate pipelines with real-world data and scenarios before deploying to production.