https://community.databricks.com/t5/administration-architecture/restricting-spark-connect-behind-premium-plan-paywall/m-p/46333#M433 <P>Quoting the <A href="https://learn.microsoft.com/en-us/azure/databricks/dev-tools/databricks-connect-ref" target="_self">databricks-connect docs</A>, "<SPAN>For Databricks Runtime 13.0 and above, Databricks Connect is now built on open-source Spark Connect." What is odd to me is that a requirement for utilizing this open source Spark feature on Databricks, is Unity Catalog, which requires a premium Databricks plan.</SPAN></P><P>I would understand if this was a restriction in databricks-connect proper, that power-users could bypass by simply using the native Spark Connect, but using Spark Connect directly to connect to a cluster outside of a Unity enabled workspace also does not work.</P><P>What is this reasoning for this restriction other than pushing users onto premium plans? Eventually non-premium/Unity users will not be able to use any form of databricks-connect once Databricks Runtime 12 is discontinued. Is there any plan for addressing this restriction?</P> Tue, 26 Sep 2023 21:13:39 GMT gandersen-codes 2023-09-26T21:13:39Z https://community.databricks.com/t5/administration-architecture/restricting-spark-connect-behind-premium-plan-paywall/m-p/46333#M433 <P>Quoting the <A href="https://learn.microsoft.com/en-us/azure/databricks/dev-tools/databricks-connect-ref" target="_self">databricks-connect docs</A>, "<SPAN>For Databricks Runtime 13.0 and above, Databricks Connect is now built on open-source Spark Connect." What is odd to me is that a requirement for utilizing this open source Spark feature on Databricks, is Unity Catalog, which requires a premium Databricks plan.</SPAN></P><P>I would understand if this was a restriction in databricks-connect proper, that power-users could bypass by simply using the native Spark Connect, but using Spark Connect directly to connect to a cluster outside of a Unity enabled workspace also does not work.</P><P>What is this reasoning for this restriction other than pushing users onto premium plans? Eventually non-premium/Unity users will not be able to use any form of databricks-connect once Databricks Runtime 12 is discontinued. Is there any plan for addressing this restriction?</P> Tue, 26 Sep 2023 21:13:39 GMT https://community.databricks.com/t5/administration-architecture/restricting-spark-connect-behind-premium-plan-paywall/m-p/46333#M433 gandersen-codes 2023-09-26T21:13:39Z