Hello @diego_poggioli , I wasn't able to find a clean solution.

One thing for sure is that we cannot use UPDATE, MERGE, stored procedures or any such statements using JDBC. We can only query, write (or overwrite) using it.

Since this is the case, I could only use ODBC. We can use pyodbc to make a connection and execute commands. However, we need ODBC driver installed for the library to work. Since, I cannot install directly on cluster or even keep init scripts (no permission for this was given 😢), I could only install inside a notebook (session) using the following code.

%sh
curl https://packages.microsoft.com/keys/microsoft.asc | apt-key add -
curl https://packages.microsoft.com/config/ubuntu/16.04/prod.list > /etc/apt/sources.list.d/mssql-release.list
sudo apt-get update
sudo ACCEPT_EULA=Y apt-get -q -y install msodbcsql17

Replace the required versions of linux and msodbc in the code. However, remember that you need to execute this command every time your cluster gets restarted or the state gets cleared.

After using this code you can use pyodbc to execute the statements. (Remember to commit() after transactions like UPDATE and close the connection after using to reduce the errors)

Hope this helps 🙂

Refer:

https://stackoverflow.com/questions/61022848/do-you-know-how-to-install-the-odbc-driver-17-for-sql-s...

https://learn.microsoft.com/en-us/sql/connect/python/pyodbc/step-3-proof-of-concept-connecting-to-sq...

https://docs.databricks.com/en/integrations/jdbc-odbc-bi.html