Seeking Best Approach for Bulk Migration of LUA/Exasol Scripts to Databricks PySpark

[Phani1]

Hi All,

We are planning a bulk migration of LUA Script / Exasol scripts to Databricks native PySpark and are evaluating the best approach for large-scale automated code conversion and testing in Databricks.

So far, we have analyzed the following options:

1. Databricks AI Assistant
   • Useful for interactive development inside notebooks / IDE
   • But it does not seem ideal for bulk migration use cases
2. Lakebridge
   • Based on the documentation, it appears to support SQL script conversion
   • It does not seem to support LUA scripts
3. External LLMs (for example Claude Opus 4.6 or similar models)
   • Calling the model through REST API from Databricks notebooks
   • Passing the source scripts for conversion
   • Writing the generated PySpark output into notebooks or files
   • Building a custom automated conversion framework

We would like to understand:

• What is the best approach for bulk conversion of LUA/Exasol scripts into PySpark on Databricks?
• Are there any Databricks-native tools, partner solutions, or migration accelerators that support this scenario?

Our goal is to find a scalable approach that supports:

• bulk conversion
• automation
• code quality review
• testing/validation in Databricks

Any recommendations, architecture suggestions, or lessons learned would be very helpful.

Thanks in advance!

---

[Ashwin_DSA · Accepted solution]

Hi @Phani1,

After some research, I don't believe there’s a Databricks-native, one-click tool to bulk-convert Lua/Exasol to PySpark.

Databricks AI Assistant is great for interactive refactoring, but as you said, it’s not really a bulk‑migration engine. Lakebridge (per your note) is geared toward SQL conversion, not Lua procedural logic, so it will only cover part of the problem. 

For this kind of migration, I’d recommend:

• Classify each script into SQL-heavy vs. Lua‑heavy logic.
• For SQL-heavy parts, use migration tooling (e.g., Lakebridge, where applicable) or an LLM to translate to Databricks / Spark SQL, then standardise and lint.
• For Lua-heavy control flow, use an LLM to generate PySpark/DBSQL skeletons, but treat that as a starting point and enforce code review + tests.
• Wrap the whole thing in a small migration factory: version control, automated conversion, and data reconciliation (parallel runs on Exasol vs Databricks to compare row counts/aggregates).

For large estates, the most effective approach is usually to combine this approach with a Databricks PSA/migration partner that already has patterns and accelerators for Exasol migrations. The best next step is to go via your Databricks account team so they can connect you with the right migration partner/PS team for your scale and timelines.

If this answer resolves your question, could you mark it as “Accept as Solution”? That helps other users quickly find the correct fix.

[Phani1]

Hi @Ashwin_DSA,

Thanks for your response—this really helps validate our approach.

We agree on the need to split SQL-heavy and LUA-heavy logic and are planning a “migration factory” with LLM-assisted conversion, followed by strong validation (parallel runs + data reconciliation) rather than relying purely on code accuracy.

We’ll also explore connecting with Databricks PS/partners as suggested

Thanks again!

Phani