topic Re: Databricks Apps with Pyodbc Microsoft SQL Driver in Data Engineering

Databricks Apps with Pyodbc Microsoft SQL Driver

bcodernet — Mon, 14 Jul 2025 19:54:19 GMT

I'm building an app that interfaces with an Azure SQL Database. I need to use Entra auth with a service principal, which is why I'm using the Microsoft ODBC driver. However, this works fine on my local, but I can't figure out how to get the ODBC drivers installed on the app cluster. I can achieve this through an init script on normal clusters, but not sure how to accomplish this on app clusters.

Re: Databricks Apps with Pyodbc Microsoft SQL Driver

szymon_dybczak — Mon, 14 Jul 2025 20:21:51 GMT

Hi @bcodernet ,

Each Databricks app can include dependencies for Python, Node.js, or both. You define these dependencies in language-specific files:

Use a requirements.txt file to specify additional Python packages.
Use a package.json file to specify Node.js packages.

Manage dependencies for a Databricks app - Azure Databricks | Microsoft Learn

Re: Databricks Apps with Pyodbc Microsoft SQL Driver

bcodernet — Mon, 14 Jul 2025 20:27:03 GMT

Hi @szymon_dybczak ,

I'm fully aware of the requirements.txt to install libraries. I have used it to install the pyodbc dependency. However, you still have to download and install Microsoft ODBC drivers and that can't be done through the requirements.txt route. On normal clusters, you have to use an init script to run a few shell commands to get this to work.

Re: Databricks Apps with Pyodbc Microsoft SQL Driver

szymon_dybczak — Mon, 14 Jul 2025 21:10:22 GMT

Sorry, you're right. But the thing is that serverless compute does not support JAR file installatioQn, so you cannot use JDBC or ODBC drivers.
But with Databricks Apps you have ability to trigger Lakeflow Jobs. So you can create a job that will use classic compute and on that compute install ODBC driver. So the ingestion part of your app will be handled by classic compute job and then you can operate on result within an app using serverless compute

Best practices for serverless compute - Azure Databricks | Microsoft Learn