Hi!
We have a lot use cases were we need to load data from Snowflake into Databricks, where users are using both R and Python for further analysis and machine learning. For this we have been using Lakehouse Federation combined with basic auth, but are now moving into using the OAuth based Snowflake integration. This has a few problems which makes it not usable in practice:
- Every user have to set up their own connection and catalog (which means that normal users need a lot rights in Databricks)
- Every user needs access to client secrets to insert it into the GUI (since SSO is a button, we can't do this with code in any way). This also have to be done every 90 days (length of the client secret from Snowflake)
- The connection object in Databricks is tightly coupled with a Snowflake role and warehouse, which means you have to create a new connection per role (makes it extremely impractical)
Does anyone have a workaround on this which makes it possible to use Lakehouse Federation and OAuth in combination? Or if anybody knows about a roadmap for the Lakehouse Federation integrations?
Appreciate any help or tips on this issue!
