Additional information I forgot to write.

Compute Instance has a User managed Identity in Azure, a Service Principal access is created in databricks with its Application ID. Same with the compute cluster, it has its own User Managed Identity that is also a SP in Databricks.

Both of them have the correct roles/rights to access clusters.

In local, I do "az login" to get my personal user, which also in databricks but has user.

In the compute cluster, I outputed the .databricks-connect file, to put it in my local computer. And tried to run the python code and it worked. I checked the Cluster logs, and I was using the Computer Cluster Service Principal which is the managed identity on Azure. So The Service Principal has the correct rights.