🚀 The Challenge
Enterprises often have hundreds or even thousands of legacy SQL queries or stored procedures written in platforms like SQL Server, Teradata, or Snowflake.
Migrating these to Databricks manually can be slow, tedious, and error-prone — each procedure needs to be re-written, validated, and tested, often involving repetitive work across teams.
In customer engagements, this migration effort repeatedly showed up as a bottleneck in modernization projects. That inspired the creation of BrickMod — a field-developed Databricks-native AI tool that makes this process faster, smarter, and more reliable.
💡 Introducing BrickMod: A tool using Databricks AI to Migrate & Modernize
BrickMod is an interactive web app that leverages Databricks AI models to automatically translate SQL procedures and business logic into Databricks-ready SQL or PySpark code.
It’s designed for both individual developers who want quick conversions and teams who need to automate migrations at scale.
🧭 How It Works
The app is divided into three main tabs — each representing a key stage in the migration journey:
🔹 1. Interactive Conversion
- Paste or upload a SQL / stored procedure directly in the UI and click Convert.
- BrickMod instantly generates the equivalent Databricks SQL, displayed side-by-side for comparison.
- A validation check automatically runs on the generated code to ensure syntax correctness. If validation fails, a “Try to Fix” button appears — it feeds the error context back to the LLM to automatically reattempt conversion and fix the issue.
This mode is perfect for testing and fine-tuning individual procedures.
🔹 2. Batch Jobs
- Upload multiple SQL files using the Select Files button
- BrickMod parallelizes conversions across Spark workers for scalability
- The results table displays the converted scripts, their status, and any issues detected
- Includes an automatic retry mechanism — if a file fails validation, BrickMod automatically reprocesses it using the latest prompt and context to fix the issue
- The final output can be generated as a Python or SQL notebook, a Databricks job, or even a stored procedure, ready to run directly in the target workspace
This mode is optimized for enterprise-scale migrations where hundreds of files must be processed efficiently.
🔹 3. Reconciliation
- Compares source and target schemas for structural or data mismatches
- Highlights discrepancies and validation errors
- Displays an LLM generated summary of the result
This ensures quality and accuracy at the tabular level.
✨ What Makes BrickMod Stand Out
-
🧠 AI-Powered Conversion – Uses LLMs to interpret and re-generate SQL logic
-
⚡ Databricks-Native – Built using Databricks Apps, DBSQL, Spark parallelism, and Delta tables
-
🧩 Configurable Prompts – LLM prompts are stored in YAML files, allowing prompt iteration without changing code
-
🌓 Dark/Light Mode – Simple toggle to personalize your workspace
-
🔁 Version-Controlled in GitHub – The entire app is maintained in a GitHub repository with full version control for collaboration and tracking
🧱 Architecture
📈 Results & Impact
In customer migrations, BrickMod has delivered significant efficiency gains. Teams have seen up to 10× faster conversion of legacy SQL scripts, while automated validation has eliminated manual review cycles. Data engineers experience instant productivity boosts, and organizations benefit from lower migration costs and shorter project timelines.
🧩 Easy to Install and Use
One of the biggest strengths of BrickMod ➜ Databricks AI is how easy it is to get started.
The GitHub repository comes with a ready-to-run Databricks notebook that handles the entire setup process automatically.
Simply open the notebook in your own Databricks workspace and run it — the notebook will install all required dependencies, deploy the BrickMod web app directly into your workspace, and launch it instantly so you can start converting SQL right away. No manual configuration, no environment setup — just run the notebook and the app is ready to use.
This makes BrickMod not only powerful, but also incredibly simple to deploy across teams or customer environments.
🧠 Model Selection and Optimization
For model selection and performance tuning, follow the official Databricks documentation on foundation model selection.
For very large batch conversions, BrickMod performs best when you use the AI Functions optimized models. These models are specifically tuned for higher throughput and lower latency, making them ideal for large-scale migrations.
💡 Best Practice for Large Stored Procedures
When converting very large stored procedures, the LLM may occasionally truncate the output if the procedure is too long to fit within model response limits.
To handle this efficiently, BrickMod allows the same logic to be converted into individual SQL cells within a Databricks notebook instead of a single stored procedure.
This approach ensures complete conversion and better readability, especially for long or complex scripts.
🏁 Conclusion
With BrickMod ➜ Databricks AI: Migrate & Modernize, teams can accelerate their modernization journey — converting complex SQL logic into Databricks-ready code in seconds.
The solution is fully version-controlled in GitHub, includes a self-installing Databricks notebook, and is designed to be both developer-friendly and enterprise-ready.
It reduces migration effort, improves reliability, and lets engineers focus on innovation rather than repetitive syntax work.
👉 Watch the demo video above to see how easily BrickMod can be deployed and used within your own Databricks workspace!
🔗 Learn More
-
GitHub Repository - https://github.com/mohanab89/databricks-migrator-with-llm
- Supported foundation models on Mosaic AI Model Serving - https://docs.databricks.com/aws/en/machine-learning/model-serving/foundation-model-overview
Credits
This project was developed by Mohana Basak with contributions from Mehran Golestaneh, who played a key role in building and testing the solution.
Disclaimer
This project and the accompanying app are not official Databricks products. They are community-built resources provided as-is, with no dedicated ongoing support. Please review and test in your environment, as they are intended for use at your own risk.
