We figured it out. Need create an Azure User Assigned Identity, and add this to the function app. The we added this user managed identity as a service principal in the Databricks workspace The following code then works

credential=ManagedIdentityCredential(client_id=USER_ASSIGNED_IDENTITY_CLIENT_ID)
token=credential.get_token("2ff814a6-3304-4ab8-85cb-cd0e6f879c1d/.default").token

with sql.connect(
   server_hostname=DATABRICKS_HOST, 
    http_path=DATABRICKS_WAREHOSE,
    access_token=token) as connection:
        with connection.cursor() as cursor:
            cursor.execute(f"SELECT 'albert' AS name")
            result = cursor.fetchall()
            df=pd.DataFrame(result, columns=[desc[0] for desc in cursor.description])

The USER_ASSIGNED_CLIENT_ID is here the Client ID of the user assigned idenity. 

Hope to be able to use the System Managed Identity in the future, to avoid the extra steps.