Hi Mark,

I have tried to use the option "ActiveDirectoryManagedIdentity", but didn't succeed.

I have downloaded the .jar files and installed them in the libraries tab of my cluster. Then I started the cluster. The cluster logfiles show that the .jar files are installed. The runtime of the cluster is "17.3 LTS (includes Apache Spark 4.0.0, Scala 2.13)"

When I run the python script below I receive the error "The authentication value "ActiveDirectoryManagedIdentity" is not valid"

 If I run the same script, but with the authentication option "ActiveDirectoryMSI" it runs without issues

I did read that it does not really matter which authentication method you use, because they are actually the same "ActiveDirectoryManagedIdentity" is the new name for Azure Managed Identity authentication instead of "ActiveDirectoryMSI".
Difference between Active Directory Managed Identity and MSI in Azure SQL Database - Microsoft Q&A

But I still wonder, why the option "ActiveDirectoryManagedIdentity" does work for you and not for me.

Kind regards,
Marco