topic Re: import pymssql fails on DLT Serverless in Data Engineering

import pymssql fails on DLT Serverless

Prashant2 — Sat, 03 May 2025 02:39:01 GMT

I have a delta live table pipeline which works fine on normal DLT job cluster.

But as soon as we switch it to use serverless compute it fails.

The failure happens at "import pymssql" after doing pip install pymssql as first statement of the source code notebook of the DLT Pipeline.

The error message is not helpful at all, but i tried to narrow it down to this statement after number of executions.

Is it a bug in DLT Serverless runtime? the same code works on serverless spark cluster as well

Re: import pymssql fails on DLT Serverless

lingareddy_Alva — Sat, 03 May 2025 18:46:47 GMT

@Prashant2

You're encountering an important limitation of Databricks DLT pipelines on Serverless Compute: currently, custom Python package installation using pip install within the notebook is not supported in DLT Serverless.
This is not a bug, but rather a limitation by design.

In Serverless DLT, the runtime environment is managed and hardened, and it does not allow arbitrary package installations at notebook execution time for:
- Security
- Stability
- Faster cold start times

While a regular serverless cluster (like for interactive notebooks or jobs) allows pip install,
DLT Serverless pipelines have restricted environments and must use packages that are:
- Pre-installed in the serverless runtime, or
- Packaged via a custom wheel or installed via a requirements.txt at deployment time (not inside the notebook)

Re: import pymssql fails on DLT Serverless

eniwoke — Sat, 03 May 2025 22:33:36 GMT

Hi @Prashant2 I am curious to know how you installed the library in your notebook. Did you use

%pip install pymssql

If so, could you try using a shell command instead, like:

!pip install pymssql

I’ve had success using !pip install in serverless compute environments + dlt serverless jobs. For example, I recently installed weasyprint this way and was able to use it in a UDF for my final Delta Live Table.

Let me know if that works for you!

Re: import pymssql fails on DLT Serverless

Prashant2 — Sun, 04 May 2025 15:29:18 GMT

Thanks for your response Eni, appreciate it.

i did try both % and !
It seems the problem is with the compatibility of pymssql library with the DLT serverless compute specifically , as i am able to install and run import other libraries from pip but not pymssql

From driver logs it appears that the pip install for pymssql succeeds successfully for fails to run the import statement due to some compatibility with the DLT serverless compute runtime

You may try it in your sample pipeline as well

Re: import pymssql fails on DLT Serverless

eniwoke — Sun, 04 May 2025 18:02:05 GMT

Hi @Prashant2, you are right, I just tried as well in my pipeline and i get an error at the import statement after installing pymssql library

If it is not too much, you can try with pyodbc, seems to work.