I have a publicly accessible SQL database that is protected by a firewall. I am trying to connect this SQL database to Databricks, but I'm encountering an authentication error. I have double-checked the credentials, port, and host, and they are all correct.

To allow the connection, I whitelisted the public IP of the NAT gateway attached to the Databricks subnet in the SQL database firewall. However, when I checked the SQL database logs, I noticed that the incoming requests are not coming from the NAT gateway's public IP. Instead, the requests are originating from the private IP of the Databricks all-purpose cluster.

Why is this happening, and how can I resolve it?