NiallEgan__Data
Databricks Employee
Databricks Employee

Hi @Borislav Blagoev​ ,

Thanks very much for taking the time to collect these logs.

The problem here (as indicated by the `IpAclValidation` message) is that IP allow listing (enabled for your workspace) will not allow arbitrary connections from Spark clusters back to the control plane by default. The subnet(s) for the data plane needs to be added to the IP allow list.

Here is a link to docs on IpACL validation: https://docs.databricks.com/security/network/ip-access-list.html

Here is a link to docs on managing subnets in the data plane: https://docs.databricks.com/administration-guide/cloud-configurations/aws/customer-managed-vpc.html#...

Not sure what your end goal is with this, but it's probably also worth mentioning that there are (better) alternatives to using the `databricks-sql-connector` on Databricks notebooks. For example, in a Python notebook you can just use `spark.sql(...)` to execute SQL commands.

View solution in original post