cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

if else conditions in databricks asset bundles

sandy311
New Contributor III

Can I use if-else conditions in databricks.yml and parameterize my asset bundles similarly to Azure Pipelines YAML?

sandeepsharma
4 REPLIES 4

szymon_dybczak
Contributor III

Hi @sandy311 ,

Yes, since databricks assets bundles are based on GO templating mechanism you can leverage that. So you can do following:
Solved: Asset Bundles : how to conditionally set content o... - Databricks Community - 77236

---
version: "1.0"
{{- if .topic_sourcing }}
status: enabled
topics:
  - name: "my-name"
    table_name: "my-tablename"
{{- else }}
status: disabled   
{{- end }}

 

sandy311
New Contributor III

If I'm not mistaken, this will only work when we initialize the Databricks asset bundle. What I need is a solution that applies conditions while validating or deploying the bundle.

sandeepsharma

Hi @sandy311 ,

Yep, you're right. It won't work with validation. 

filipniziol
Contributor

Hi @sandy311 ,

Could you please provide more details on what youโ€™re trying to achieve?

It seems like you are looking to use Databricks Asset Bundles as complete CI/CD pipelines. While Databricks Asset Bundles are a crucial part of the CI/CD process, they focus on deploying Databricks resources like notebooks, wheels, and jobs. They do not inherently provide the full capabilities of a CI/CD pipeline such as automated testing, conditional deployments, or rollback mechanisms.

For example, when you mentioned in your previous question about deploying Databricks jobs, validating their execution, and deciding to retain or roll back the deployed code based on the jobโ€™s success, this type of workflow is typically managed by a CI/CD platform such as Azure DevOps, GitLab, or GitHub Actions. These platforms allow for advanced control over deployment, testing, and error handling that extends beyond the capabilities of Databricks Asset Bundles alone.

Key Points:

  1. Integration with CI/CD Pipelines: Databricks Asset Bundles are used within CI/CD pipelines but do not replace them. They serve as a mechanism for defining and deploying Databricks resources consistently across environments.

  2. CI/CD Platforms for Workflow Management: For tasks such as conditional deployments, testing job outcomes, and performing rollbacks, you should leverage CI/CD tools like Azure DevOps, GitLab CI/CD, or GitHub Actions. These tools provide the necessary logic and controls to manage complex workflows and automate decision-making processes.

  3. Deployment Control: While Databricks Asset Bundles can include some level of parameterization and conditional logic during initialization (using Go templating or variables), the actual control over validation, testing, and deployment flows is best handled by an external CI/CD tool.

Summary:

Databricks Asset Bundles are designed for deploying Databricks assets as part of a broader CI/CD strategy. To achieve end-to-end CI/CD functionalities such as testing, validation, and rollback, these bundles should be integrated with a dedicated CI/CD platform.

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโ€™t want to miss the chance to attend and share knowledge.

If there isnโ€™t a group near you, start one and help create a community that brings people together.

Request a New Group