Hi @tuckera
The data engineering landscape shows a pretty diverse mix of approaches for tracking and deploying pipelines and data assets, often varying by company size, maturity, and specific needs.
Infrastructure as Code (IaC) tools like Terraform and Pulumi are increasingly popular for the infrastructure layer - managing compute resources, storage, networking, and cloud services. Many teams use these for the foundational infrastructure while layering other tools on top for pipeline-specific concerns.
Platform-specific deployment tools are common too. For Databricks environments, DABs (Databricks Asset Bundles) are gaining traction since they provide native integration and handle both infrastructure and code deployment. Similarly, teams on AWS often use CDK or CloudFormation, while Azure teams might use ARM templates or Bicep.
CI/CD platforms like GitHub Actions, GitLab CI, Jenkins, or Azure DevOps handle the orchestration layer - testing, building, and deploying changes. These often coordinate with the IaC tools and handle secrets management, environment promotion, and rollback capabilities.
Data-specific orchestration tools like Airflow, Prefect, Dagster, or managed services like Azure Data Factory handle pipeline scheduling and dependency management. Some of these also provide deployment and versioning capabilities for the pipelines themselves.
Hybrid approaches are probably most common in practice - using Terraform for infrastructure, CI/CD for deployment orchestration, and platform-specific tools for application-layer concerns. Many teams also maintain some manual processes for sensitive production changes or complex migrations.
The "right" approach often depends on your existing tech stack, team expertise, compliance requirements, and how much you value vendor lock-in versus native integration.
LR
