Databricks Apps with Pyodbc Microsoft SQL Driver

[bcodernet]

I'm building an app that interfaces with an Azure SQL Database. I need to use Entra auth with a service principal, which is why I'm using the Microsoft ODBC driver. However, this works fine on my local, but I can't figure out how to get the ODBC drivers installed on the app cluster. I can achieve this through an init script on normal clusters, but not sure how to accomplish this on app clusters.

[szymon_dybczak]

Hi @bcodernet ,

Each Databricks app can include dependencies for Python, Node.js, or both. You define these dependencies in language-specific files:

• Use a requirements.txt file to specify additional Python packages.
• Use a package.json file to specify Node.js packages.

Manage dependencies for a Databricks app - Azure Databricks | Microsoft Learn

[bcodernet]

Hi @szymon_dybczak ,

I'm fully aware of the requirements.txt to install libraries. I have used it to install the pyodbc dependency. However, you still have to download and install Microsoft ODBC drivers and that can't be done through the requirements.txt route. On normal clusters, you have to use an init script to run a few shell commands to get this to work.

[szymon_dybczak]

Sorry, you're right. But the thing is that serverless compute does not support JAR file installatioQn, so you cannot use JDBC or ODBC drivers.
But with Databricks Apps you have ability to trigger Lakeflow Jobs. So you can create a job that will use classic compute and on that compute install ODBC driver. So the ingestion part of your app will be handled by classic compute job and then you can operate on result within an app using serverless compute

Best practices for serverless compute - Azure Databricks | Microsoft Learn