Quoting the databricks-connect docs, "For Databricks Runtime 13.0 and above, Databricks Connect is now built on open-source Spark Connect." What is odd to me is that a requirement for utilizing this open source Spark feature on Databricks, is Unity Catalog, which requires a premium Databricks plan.
I would understand if this was a restriction in databricks-connect proper, that power-users could bypass by simply using the native Spark Connect, but using Spark Connect directly to connect to a cluster outside of a Unity enabled workspace also does not work.
What is this reasoning for this restriction other than pushing users onto premium plans? Eventually non-premium/Unity users will not be able to use any form of databricks-connect once Databricks Runtime 12 is discontinued. Is there any plan for addressing this restriction?
