Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
Showing results for 
Search instead for 
Did you mean: 

Brickflow - An opinionated python framework to help build and deploy Databricks workflows at scale

New Contributor II
Hello, Databricks Community,
I’m thrilled to introduce “Brickflow,” our innovative solution designed to streamline the development and deployment of workflows at scale, leveraging straightforward Python coding. This tool has already achieved a significant milestone with over 500,000 downloads on PyPI since its inception. You can explore Brickflow further at docs and download from PyPI - Brickflows.
Key Features of Brickflow:
  • Integration with Databricks Asset Bundles (DABs): Brickflow seamlessly deploys your workflows using DABs, optimizing performance and scalability.
  • Airflow Compatibility: For those transitioning from Airflow or familiar with its operations, Brickflow offers a smooth migration path with its decorator pattern for task creation, making it incredibly intuitive.
  • Simplicity and Flexibility: Write your tasks as plain Python functions, with built-in support to access Spark and SQL within the Brickflow context, enhancing efficiency and ease of use.
  • Task Dependencies and Workflow Management: Easily define dependencies between tasks, supporting complex workflow orchestration.
  • Extended Task Parameters: Brickflow introduces a set of common task parameters, enriching your workflows with additional functionality and control. This also helps with your audit reports and dashboard for cost metrics etc.
  • Airflow Integration for Seamless Migration: Utilize Brickflow’s TaskDependencySensor feature to maintain task dependencies on Airflow dags, ensuring uninterrupted workflow integration during the migration process.
  • Powerful CLI for Deployment: Leverage the Brickflow CLI for efficient project deployment across multiple environments. The “–env” option facilitates deployment scalability and integration with CI/CD pipelines.
  • Local Environment Support: Engineers can develop and test workflows in a ‘local’ environment, promoting collaboration without disrupting ongoing work.
Brickflow is designed to empower engineers to focus more on solving business problems and less on the intricacies of workflow management. We’re eager to hear your thoughts and feedback as we continuously strive to enhance Brickflow’s capabilities.
Please give it a try, and let us know your experiences. Your insights are invaluable to us.
Thank you for considering Brickflow for your workflow automation needs. We look forward to your feedback and contributions to making Brickflow even better.
#DatabricksWorkflows #Workflows #Brickflows
Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!