cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Genie Hub
Explore technical articles, practical guides, best practices, and real-world use cases to help you get the most out of Databricks Genie. Learn from the Databricks team, MVPs, and community experts.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

Best Practices required in building the projects

ItsTechieeGirl
New Contributor

Hi People,

I am working on an important project, can anyone guide me with the best practices which can be implemented in the projects with databricks, sql, python integrated projects.

 

with regards,

Manasa BR

Data Analyst

3 REPLIES 3

data_pulse
Contributor

@ItsTechieeGirl 

For projects integrating Databricks, SQL, and Python, there are several established best practices worth considering. While the overall approach depends on the use case, data sources, workload and project requirements, following some common engineering principles can help build scalable and production ready solutions.

A good starting point would be to explore Databricks official developer guidance and articles, which cover different aspects of project architecture, development, testing and deployment.

Some general best practices to consider:

  • Project structure and version control: Organize SQL scripts, Python modules, notebooks, and configuration files in a structured Git repository. Avoid relying entirely on workspace notebooks for production code.
  • Modular development: Use SQL for declarative transformations and Python/PySpark for reusable processing logic, integrations, and automation where appropriate.
  • Deployment and CI/CD: Consider Databricks Declarative Automation Bundles(DAB) to manage jobs, pipelines, and environment-specific configurations consistently.
  • Testing and data quality: Implement unit tests using pytest, SQL-based validation and data-quality checks within pipelines.
  • Governance and operations: Leverage Unity Catalog for access control and governance, along with appropriate monitoring, logging, and error-handling practices.

One other approach is to Accelerate development with Genie Code and AI agents: Leverage Genie Code or AI coding agents to quickly scaffold project structures, Git repositories, SQL/Python modules, Lakeflow pipelines, Databricks Bundles, and initial tests. This reduces boilerplate effort and speeds up development.

Useful resources to explore:

Official Databricks docs:

Developer Best Practices

Lakeflow Pipeline Best Practices

CI/CD on Databricks

Databricks Community articles:

How to Build, Test, and Ship SQL Pipelines on Databricks

Productionizing Databricks Pipelines with Declarative Automation Bundles and Azure DevOps

These are excellent references. 

From your post, it sounds like the tools (Databricks, sql, python) are selected. These are popular and powerful tools.

To start a new project, it is still worth considering the final products, those that you will show them to make your client happy, and the input, data is definitely part of the input. Plan and think through carefully to the structure of your tables at each storage layer. Do prototypes and discuss to your clients.
End users rarely seriously care how beautifully the code is written, they happy when they get what they asked quickly. However, clean and easy to change code will help us to deliver faster in the long run. So do all the best practices. 

 

balajij8
Esteemed Contributor III

@ItsTechieeGirl When setting up an integrated Python and SQL project on Databricks, you need to first establish clean code structure and modularity by treating notebooks as thin orchestration layers. Place the Python functions into importable .py modules under src/proj/py and keep SQL queries in .sql files under src/proj/sql/, referencing both from the jobs and pipelines. You should version control everything in a single repository including notebooks, source files and DAB configs (databricks.yml and environment overrides) but excluding build artifacts, secrets and PII via .gitignore. Using DAB gives you unified deployment and testing and you can orchestrate production workflows with Lakeflow Jobs. 

For data pipelines, I recommend using the medallion architecture across three layers separating ingestion and transformation into distinct pipelines. Each layer can be scheduled and troubleshooted independently and a downstream transformation failure won't block new bronze data from landing. In the bronze layer, ingest raw data via streaming tables with minimal transformation and use external tables to preserve the raw data. In silver layer, you can then apply incremental cleaning, filtering and enrichment via streaming tables and materialized views, while the gold layer will be for business aggregations and metrics in materialized views for dashboards and reporting. Organize the layers with Unity Catalog schemas (like sales.bronze_transactions) and use managed tables with schema enforcement in silver and gold. At the table level, enable delta.enableChangeDataFeed, apply liquid clustering to large tables and enable predictive optimization. To handle testing, CI/CD and production readiness, implement three testing layers that match the deployment path.

More details here