Thanks for the reply It is very useful and comprehensive.

I managed to find another solution to the problem so I wanted to share some additional details on this topic:
I was using 15.4 LTS ML Runtime, this could have caused the problem - I did not switch to 16.4 since that might have other breaking changes on other parts of my project. 

My solution was: I realized that in the container created by Databricks for the serving endpoint behind the scenes it installs a package 'databricks-feature-lookup==1.*' inside the container and for my usecase it defaulted to version 1.4 which is an older version. Instead for the automatic authentication to Online tables to work 'databricks-feature-lookup==1.9' is required. -> I have not found any documentation that mentioned this dependency so I wanted to share this - it might be useful to mention this somewhere in the docs as it was a bit hidden and hard to debug.

By packaging the model with this dependency fixed inside the FeatureEngineeringClient.log_model() we can ensure that the right version of this package gets installed inside the serving-endpoint's docker and we won't get the auth error again.