Snowflake to Databricks migration

[tarunnagpal]

We are working on a proposal for our existing customer to migrate approximately 500 tables and the associated business logic from Snowflake to Databricks. The business logic is currently implemented using stored procedures, which need to be converted into SparkSQL on Databricks. 

Can anyone please share any relevant insights, tools, or best practices that might help streamline this transition?

[MariuszK]

You can write a custom parser that will translate it. SQL migration is simple, you use UDF to cover not existing functions. Challenging can be procedure translation, it will depends on language that you use in store procedures?

Model and data migration will be easiest part, the model can be translated to Databricks SQL, data extracted to delta format or parquet.

LLMs can streamline translation, but you need to test it and review.

[sunnydata]

Hi @tarunnagpal !!

Adding to what @MariuszK said,

1. Using an LLM to accelerate the translation process is a great approach, but if the code is proprietary, it's best to use a closed model.

2. Implementing a validation process is crucial to ensure that the translated tables in Databricks match the originals in Snowflake.

3. Since Databricks doesn’t support stored procedures, you can replace them with Notebooks orchestrated by workflows or Delta Live Tables (DLTs). While DLTs may require more translation effort upfront, they can offer long-term benefits.

At SunnyData, we’ve developed solutions to streamline steps 1 and 2:

• We’ve been customizing an LLM to improve translation accuracy for these types of migrations. While we haven’t deployed it yet, we’re actively exploring its capabilities.
• We’ve built a solution that gathers statistical insights on tables and performs large-scale comparisons to validate their equivalence, even with massive datasets. It also highlights any discrepancies.

Would love to hear more about your specific migration challenges! Feel free to DM me for any follow-up questions!

Best, 

Eliana Oviedo
BD & Partnerships | Strategist

E: eliana.oviedo@sunnydata.ai

LinkedIn