article Azure Databricks — Reverse SSH Tunnel for On-Premises Connectivity in Technical Blog

article Azure Databricks — Reverse SSH Tunnel for On-Premises Connectivity in Technical Blog https://community.databricks.com/t5/technical-blog/azure-databricks-reverse-ssh-tunnel-for-on-premises-connectivity/ba-p/155969 <H2 id="b3b7" class="acv acw xg bb acx ns acy nt nu nv acz nw nx ada adb adc add ade adf adg adh adi adj adk adl adm bg" data-selectable-paragraph="">Introduction</H2> <P class="pw-post-body-paragraph abx aby xg abz b aca adn acc acd ace ado acg ach aci adp ack acl acm adq aco acp acq adr acs act acu vi bg" data-selectable-paragraph="">Azure Databricks users often need to access on-premises resources, such as databases, that reside in their corporate networks. In most cases, the right network path, like ExpressRoute or a Site-to-Site VPN combined with a private endpoint or a load-balanced proxy, is enough to get traffic through. However, in some enterprise environments, the on-premises database sits behind a corporate firewall that blocks all inbound traffic, and the database or network team cannot open inbound ports, even for internal traffic. This is where most of the standard patterns stop working.</P> <P class="pw-post-body-paragraph abx aby xg abz b aca acb acc acd ace acf acg ach aci acj ack acl acm acn aco acp acq acr acs act acu vi bg" data-selectable-paragraph="">Previously, we published blogs on<SPAN> </SPAN><A class="z nh" href="https://medium.com/databricks-platform-sme/azure-databricks-serverless-private-connectivity-to-customers-resources-part-2-b1c43ed8fcd4" target="_blank" rel="noopener" data-discover="true">Standard Load Balancer based connectivity</A><SPAN> </SPAN>and<SPAN> </SPAN><A class="z nh" href="https://medium.com/databricks-platform-sme/azure-databricks-serverless-private-connectivity-to-customer-resources-through-azure-application-32ea023b2b26" target="_blank" rel="noopener" data-discover="true">Azure Application Gateway based connectivity</A><SPAN> </SPAN>for Azure Databricks Serverless. Both assume a reachable target in the customer’s Azure tenant or a reachable on-premises target over the existing network connectivity. In this post, we look at a different model, a reverse SSH tunnel proxy hub that works when inbound access to the on-premises network is not allowed.</P> <P class="pw-post-body-paragraph abx aby xg abz b aca acb acc acd ace acf acg ach aci acj ack acl acm acn aco acp acq acr acs act acu vi bg" data-selectable-paragraph="">In this blog, we walk through the architecture and present options to build a reverse-tunnel proxy hub with high availability. We will also show how this new connectivity pattern supports both Azure Databricks classic and serverless compute, without relaxing any on-premises inbound firewall restrictions. This is particularly useful for Lakeflow Connect customers ingesting from on-premises databases, where the connection must initiate from the customer’s network.</P> <BLOCKQUOTE class="ads adt adu"> <P class="abx aby adv abz b aca acb acc acd ace acf acg ach aci acj ack acl acm acn aco acp acq acr acs act acu vi bg" data-selectable-paragraph=""><STRONG class="abz pf">Note:</STRONG><SPAN> </SPAN>We use MySQL on port 3306 as the running example throughout this post. The same pattern works for any TCP-based database — PostgreSQL on 5432, Oracle on 1521, SQL Server on 1433, and so on.</P> </BLOCKQUOTE> <P class="acv acw xg bb acx ns acy nt nu nv acz nw nx ada adb adc add ade adf adg adh adi adj adk adl adm bg"> </P> <H2 id="3b9e" class="acv acw xg bb acx ns acy nt nu nv acz nw nx ada adb adc add ade adf adg adh adi adj adk adl adm bg" data-selectable-paragraph="">Why a reverse SSH tunnel?</H2> <P class="pw-post-body-paragraph abx aby xg abz b aca adn acc acd ace ado acg ach aci adp ack acl acm adq aco acp acq adr acs act acu vi bg" data-selectable-paragraph="">In most enterprise networks, outbound traffic from on-premises to the cloud is permitted over ExpressRoute, a Site-to-Site VPN, or SD-WAN, while inbound traffic from the cloud into the on-premises network is restricted by the corporate firewall. A conventional connectivity model, where Azure Databricks initiates the connection to the on-premises database, requires inbound firewall rules to be configured. This may not be allowed in some environments, or may require a compliance review. A reverse SSH tunnel inverts this flow — the on-premises side initiates the connection, dialing out to a proxy virtual machine (VM) in the cloud over SSH. The proxy VM only needs to accept an outbound SSH connection, which the corporate firewall already permits.</P> <P class="pw-post-body-paragraph abx aby xg abz b aca acb acc acd ace acf acg ach aci acj ack acl acm acn aco acp acq acr acs act acu vi bg" data-selectable-paragraph="">The word “reverse” can be misleading. The SSH connection itself is a normal outbound connection from on-premises to the cloud. What is “reverse” is the direction of the application traffic that rides on top of it. When the on-premises host runs ssh -R against the proxy VM, it opens a listener on the proxy that forwards any incoming connection back over the SSH session to the on-premises database. Azure Databricks, sitting on the same VNet as the proxy VM, connects to that listener and reaches the database as if it were local to the cloud.</P> <P class="pw-post-body-paragraph abx aby xg abz b aca acb acc acd ace acf acg ach aci acj ack acl acm acn aco acp acq acr acs act acu vi bg" data-selectable-paragraph="">This direction aligns cleanly with the firewall policy in most enterprises. Outbound SSH (port 22) from on-premises to the cloud is almost always permitted for internal traffic. Inbound SSH is almost never permitted. The reverse tunnel takes advantage of this asymmetry and gives us a secure, authenticated path for database traffic without relaxing any firewall restrictions.</P> <P class="acv acw xg bb acx ns acy nt nu nv acz nw nx ada adb adc add ade adf adg adh adi adj adk adl adm bg"> </P> <H2 id="d1f9" class="acv acw xg bb acx ns acy nt nu nv acz nw nx ada adb adc add ade adf adg adh adi adj adk adl adm bg" data-selectable-paragraph="">When to use this?</H2> <P class="pw-post-body-paragraph abx aby xg abz b aca adn acc acd ace ado acg ach aci adp ack acl acm adq aco acp acq adr acs act acu vi bg" data-selectable-paragraph="">The reverse SSH tunnel pattern fills a specific gap in the connectivity options for Azure Databricks. This pattern is the right fit when:</P> <UL class=""> <LI id="0417" class="abx aby xg abz b aca acb acc acd ace acf acg ach aci acj ack acl acm acn aco acp acq acr acs act acu adw adx ady bg" data-selectable-paragraph="">Your on-premises target is reachable from Azure over an existing network path (ExpressRoute, Site-to-Site VPN, or SD-WAN), but the corporate firewall blocks inbound traffic from the cloud.</LI> </UL> <P class="pw-post-body-paragraph abx aby xg abz b aca acb acc acd ace acf acg ach aci acj ack acl acm acn aco acp acq acr acs act acu vi bg" data-selectable-paragraph="">This pattern is<SPAN> </SPAN><STRONG class="abz pf">not</STRONG><SPAN> </SPAN>the right fit when:</P> <UL class=""> <LI id="3275" class="abx aby xg abz b aca acb acc acd ace acf acg ach aci acj ack acl acm acn aco acp acq acr acs act acu adw adx ady bg" data-selectable-paragraph="">Your on-premises target is reachable, and inbound access is already permitted. Simpler patterns (private endpoint, load-balanced proxy, or direct connection) cover the case more directly.</LI> <LI id="3cc1" class="abx aby xg abz b aca adz acc acd ace aea acg ach aci aeb ack acl acm aec aco acp acq aed acs act acu adw adx ady bg" data-selectable-paragraph="">No network path exists between Azure and your on-premises network. Establish ExpressRoute, VPN, or SD-WAN connectivity first before considering any cloud-to-on-premises pattern.</LI> </UL> <P class="acv acw xg bb acx ns acy nt nu nv acz nw nx ada adb adc add ade adf adg adh adi adj adk adl adm bg"> </P> <H2 id="7bca" class="acv acw xg bb acx ns acy nt nu nv acz nw nx ada adb adc add ade adf adg adh adi adj adk adl adm bg" data-selectable-paragraph="">Prerequisites</H2> <TABLE> <TBODY> <TR> <TD> <P><STRONG>Component</STRONG></P> </TD> <TD> <P><STRONG>Notes</STRONG></P> </TD> </TR> <TR> <TD> <P><SPAN>On-premises Tunnel host</SPAN></P> </TD> <TD> <P><SPAN>Runs “autossh”. Needs outbound port 22 to the proxy VMs and network access to the on-premises database.</SPAN></P> </TD> </TR> <TR> <TD> <P><SPAN>Proxy Hub VNet in Azure</SPAN></P> </TD> <TD> <P><SPAN>Connectivity to on-premises via ExpressRoute, Site-to-Site VPN, or SD-WAN.</SPAN></P> </TD> </TR> <TR> <TD> <P><SPAN>Proxy VMs in the Proxy Hub VNet</SPAN></P> </TD> <TD> <P><SPAN>At least one; two for high availability. Run “socat” and the HTTP health check.</SPAN></P> </TD> </TR> <TR> <TD> <P><SPAN>Standard Load Balancer</SPAN></P> </TD> <TD> <P><SPAN>Required for high availability (multi-VM). Optional for development or single-VM setups.</SPAN></P> </TD> </TR> <TR> <TD> <P><SPAN>Network path to Databricks</SPAN></P> </TD> <TD> <P><SPAN>VNet peering (classic compute) or NCC + Private Endpoint + Private Link Service (serverless compute).</SPAN></P> </TD> </TR> </TBODY> </TABLE> <P class="acv acw xg bb acx ns acy nt nu nv acz nw nx ada adb adc add ade adf adg adh adi adj adk adl adm bg"> </P> <H2 id="93dd" class="acv acw xg bb acx ns acy nt nu nv acz nw nx ada adb adc add ade adf adg adh adi adj adk adl adm bg" data-selectable-paragraph="">Architecture</H2> <P class="pw-post-body-paragraph abx aby xg abz b aca adn acc acd ace ado acg ach aci adp ack acl acm adq aco acp acq adr acs act acu vi bg" data-selectable-paragraph="">The solution consists of three zones — the on-premises network, the proxy hub VNet in Azure, and the Azure Databricks compute plane. Traffic flows from Databricks, through the proxy hub, and over the reverse SSH tunnels into the on-premises database.</P> <P class="pw-post-body-paragraph abx aby xg abz b aca adn acc acd ace ado acg ach aci adp ack acl acm adq aco acp acq adr acs act acu vi bg" data-selectable-paragraph=""><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="ReverseTunnel-ArchDiagram.png" style="width: 999px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/26572iB214C89AC136B3A8/image-size/large?v=v2&px=999" role="button" title="ReverseTunnel-ArchDiagram.png" alt="ReverseTunnel-ArchDiagram.png" /></span></P> <P class="pw-post-body-paragraph abx aby xg abz b aca acb acc acd ace acf acg ach aci acj ack acl acm acn aco acp acq acr acs act acu vi bg" data-selectable-paragraph="">The proxy hub VNet in the middle is the core segment or zone of this architecture. We will mostly focus on building out the following components:</P> <UL class=""> <LI id="f731" class="abx aby xg abz b aca acb acc acd ace acf acg ach aci acj ack acl acm acn aco acp acq acr acs act acu adw adx ady bg" data-selectable-paragraph=""><STRONG class="abz pf">Proxy VMs (two or more for high availability)</STRONG>. Each proxy VM accepts an incoming reverse SSH connection from the on-premises tunnel host and runs two additional services:<BR />— socat, which bridges the VM’s network interface to the tunnel listener<BR />— A lightweight HTTP health check endpoint that the load balancer probes.</LI> <LI id="d108" class="abx aby xg abz b aca adz acc acd ace aea acg ach aci aeb ack acl acm aec aco acp acq aed acs act acu adw adx ady bg" data-selectable-paragraph=""><STRONG class="abz pf">Standard Load Balancer</STRONG>. An internal Standard Load Balancer (SLB) sits in front of the proxy VMs. It provides a stable frontend IP, distributes traffic across the backend pool, and removes unhealthy VMs when the HTTP health probe fails. This is what gives us automatic failover when a tunnel fails.</LI> <LI id="0f94" class="abx aby xg abz b aca adz acc acd ace aea acg ach aci aeb ack acl acm aec aco acp acq aed acs act acu adw adx ady bg" data-selectable-paragraph=""><STRONG class="abz pf">Private Link Service (PLS)</STRONG>. The PLS exposes the SLB frontend to Azure Databricks serverless compute. Serverless compute lives in a Databricks-managed VNet and cannot use VNet peering to reach customer resources, so it needs the PLS combined with a Private Endpoint (PE) rule in a Network Connectivity Configuration (NCC) on the Databricks side.</LI> </UL> <P class="pw-post-body-paragraph abx aby xg abz b aca acb acc acd ace acf acg ach aci acj ack acl acm acn aco acp acq acr acs act acu vi bg" data-selectable-paragraph="">On the on-premises side, a single Linux host (physical or virtual) establishes the reverse tunnels. This tunnel host runs one autossh process per proxy VM, and each autossh process carries one or more -R port forwards that map a high port on the proxy VM to the on-premises database. The tunnel host does not need anything special beyond network access to the database and outbound SSH to the proxy VMs.</P> <P class="pw-post-body-paragraph abx aby xg abz b aca acb acc acd ace acf acg ach aci acj ack acl acm acn aco acp acq acr acs act acu vi bg" data-selectable-paragraph="">On the Azure Databricks side, classic compute reaches the SLB frontend through VNet peering, and serverless compute reaches it through the NCC Private Endpoint that targets the PLS. Both paths converge on the same SLB, so we serve both compute types with one proxy hub.</P> <P> </P> <H2 id="3613" class="acv acw xg bb acx ns acy nt nu nv acz nw nx ada adb adc add ade adf adg adh adi adj adk adl adm bg" data-selectable-paragraph="">How to make the tunnel reachable?</H2> <P class="pw-post-body-paragraph abx aby xg abz b aca adn acc acd ace ado acg ach aci adp ack acl acm adq aco acp acq adr acs act acu vi bg" data-selectable-paragraph="">By default, ssh -R binds its listener on the remote side to the loopback interface (<EM class="adv">localhost:13306</EM>). This is controlled by the<SPAN> </SPAN><EM class="adv">GatewayPorts</EM><SPAN> </SPAN>setting in<SPAN> </SPAN><EM class="adv">sshd_config</EM>, which ships as no on most Linux distributions. The default is deliberate — a reverse-forwarded port is not automatically exposed to the rest of the network. In our tests, we keep this default, which means the tunnel sits safely on the proxy VM’s loopback and is not reachable directly from the NIC.</P> <P class="pw-post-body-paragraph abx aby xg abz b aca acb acc acd ace acf acg ach aci acj ack acl acm acn aco acp acq acr acs act acu vi bg" data-selectable-paragraph="">So, for the default model, we need a way to bridge traffic from the NIC to the tunnel on the loopback interface. That is where socat comes in. On each proxy VM, we run a small socat process that listens on the network interface (<EM class="adv">0.0.0.0:3306</EM>) and forwards whatever it receives to<SPAN> </SPAN><EM class="adv">localhost:13306</EM>, where the tunnel is waiting. Traffic from Databricks flows into the NIC, through socat, into the tunnel, and out to the on-premises database.</P> <BLOCKQUOTE class="ads adt adu"> <P class="abx aby adv abz b aca acb acc acd ace acf acg ach aci acj ack acl acm acn aco acp acq acr acs act acu vi bg" data-selectable-paragraph=""><STRONG class="abz pf">Note:</STRONG><SPAN> </SPAN>You can skip socat by setting GatewayPorts yes on the proxy VM’s SSH server and binding the -R forward to 0.0.0.0 directly. The tunnel then listens on the NIC itself. Compared to the default model, this removes the loopback boundary as an extra defense-in-depth layer — access control falls entirely on NSG rules, source IP filtering, and database authentication. We keep the sshd default and let socat handle the bridge explicitly.</P> </BLOCKQUOTE> <P class="acv acw xg bb acx ns acy nt nu nv acz nw nx ada adb adc add ade adf adg adh adi adj adk adl adm bg"> </P> <H2 id="9a4b" class="acv acw xg bb acx ns acy nt nu nv acz nw nx ada adb adc add ade adf adg adh adi adj adk adl adm bg" data-selectable-paragraph="">How to make the tunnel resilient?</H2> <P class="pw-post-body-paragraph abx aby xg abz b aca adn acc acd ace ado acg ach aci adp ack acl acm adq aco acp acq adr acs act acu vi bg" data-selectable-paragraph="">With two or more proxy VMs behind the SLB, we already get redundancy for many common failures. If a proxy VM crashes or becomes unreachable, a simple TCP health probe lets the SLB detect it and stop sending traffic to it, and the remaining VM keeps serving. But what if just the SSH tunnel on a VM dies, while the VM itself and the socat service on it stay healthy? In that case, a plain TCP health probe on port<SPAN> </SPAN><EM class="adv">3306</EM><SPAN> </SPAN>detects that socat is listening and marks the VM as healthy, even though no traffic actually reaches the database. Databricks connections start failing intermittently while the SLB reports the VM as healthy.</P> <P class="pw-post-body-paragraph abx aby xg abz b aca acb acc acd ace acf acg ach aci acj ack acl acm acn aco acp acq acr acs act acu vi bg" data-selectable-paragraph="">To overcome this challenge and detect that state, we run a small HTTP health check service on each proxy VM. The HTTP service listens on port 8080, and for every incoming request, it attempts a TCP connection to the tunnel port on loopback (<EM class="adv">localhost:13306</EM>). If the connection succeeds, the tunnel is alive, and we return<SPAN> </SPAN><EM class="adv">HTTP 200</EM>. If the connection fails, the tunnel is dead, and we return<SPAN> </SPAN><EM class="adv">HTTP 503</EM>. The SLB is configured to probe this HTTP endpoint instead of probing port 3306 directly.</P> <P class="pw-post-body-paragraph abx aby xg abz b aca acb acc acd ace acf acg ach aci acj ack acl acm acn aco acp acq acr acs act acu vi bg" data-selectable-paragraph="">With this setup, the SLB reflects the actual health of the tunnel, not just the listener in front of it. When a tunnel dies on one proxy VM, the health probe starts returning<SPAN> </SPAN><EM class="adv">503</EM>, the SLB removes the VM from rotation within a few seconds (about 10), and traffic flows entirely through the remaining VM. When the tunnel recovers (autossh reconnects), the probe returns<SPAN> </SPAN><EM class="adv">200</EM><SPAN> </SPAN>again, and the VM rejoins the pool automatically.</P> <BLOCKQUOTE class="ads adt adu"> <P class="abx aby adv abz b aca acb acc acd ace acf acg ach aci acj ack acl acm acn aco acp acq acr acs act acu vi bg" data-selectable-paragraph=""><STRONG class="abz pf">Note:</STRONG><SPAN> </SPAN>SLB failover redirects new connections only. Queries already in flight on the failed proxy VM may be dropped and need to be retried by the Databricks client (JDBC driver, foreign catalog query, or Lakeflow Connect gateway). Once retried, the connection is routed to the healthy VM.</P> </BLOCKQUOTE> <P class="acv acw xg bb acx ns acy nt nu nv acz nw nx ada adb adc add ade adf adg adh adi adj adk adl adm bg"> </P> <H2 id="b2d8" class="acv acw xg bb acx ns acy nt nu nv acz nw nx ada adb adc add ade adf adg adh adi adj adk adl adm bg" data-selectable-paragraph="">Building the tunnel</H2> <P class="pw-post-body-paragraph abx aby xg abz b aca adn acc acd ace ado acg ach aci adp ack acl acm adq aco acp acq adr acs act acu vi bg" data-selectable-paragraph="">With the proxy hub designed, we now build the tunnel itself. This has two ends — the cloud side, where each proxy VM accepts the reverse SSH connection, and the on-premises side, where the tunnel host initiates the connection out to the cloud. We walk through both in turn.</P> <BLOCKQUOTE class="ads adt adu"> <P class="abx aby adv abz b aca acb acc acd ace acf acg ach aci acj ack acl acm acn aco acp acq acr acs act acu vi bg" data-selectable-paragraph=""><STRONG class="abz pf">Note:</STRONG><SPAN> </SPAN>The commands and sample scripts/configs in this section assume Ubuntu. Adapt package manager commands, file paths, service names, and parameters as needed for your environment.</P> </BLOCKQUOTE> <H3 id="d8c1" class="aer acw xg bb acx aes aet aeu nu aev aew aex nx aci aey aez afa acm afb afc afd acq afe aff afg afh bg" data-selectable-paragraph="">Cloud side — the proxy VM</H3> <P class="pw-post-body-paragraph abx aby xg abz b aca adn acc acd ace ado acg ach aci adp ack acl acm adq aco acp acq adr acs act acu vi bg" data-selectable-paragraph="">Each proxy VM runs a stock Linux OS with three pieces added — the socat service that bridges the NIC to the tunnel, a small Python-based HTTP health check that the SLB probes, and no changes to the default<SPAN> </SPAN><EM class="adv">sshd</EM><SPAN> </SPAN>configuration.</P> <P><STRONG>1.</STRONG> Install socat from the package repository:</P> <LI-CODE lang="python">sudo apt-get install -y socat</LI-CODE> <P class="pw-post-body-paragraph abx aby xg abz b aca acb acc acd ace acf acg ach aci acj ack acl acm acn aco acp acq acr acs act acu vi bg" data-selectable-paragraph=""><STRONG class="abz pf">2.</STRONG><SPAN> </SPAN>A sample systemd unit for socat (runs at boot and restarts if it ever exits):</P> <LI-CODE lang="python"># /etc/systemd/system/socat-db-proxy.service [Unit] Description=socat DB proxy (NIC to SSH tunnel) After=network.target [Service] ExecStart=/usr/bin/socat TCP-LISTEN:3306,bind=0.0.0.0,fork,reuseaddr TCP:localhost:13306 Restart=always RestartSec=5 [Install] WantedBy=multi-user.target</LI-CODE> <P class="pw-post-body-paragraph abx aby xg abz b aca acb acc acd ace acf acg ach aci acj ack acl acm acn aco acp acq acr acs act acu vi bg" data-selectable-paragraph="">The<SPAN> </SPAN><EM class="adv">TCP-LISTEN:3306,bind=0.0.0.0,fork,reuseaddr<SPAN> </SPAN></EM>binds on port 3306 across all network interfaces, handles each incoming connection in a separate child process, and allows the port to be reused quickly if the service restarts.<SPAN> </SPAN><EM class="adv">TCP:localhost:13306</EM><SPAN> </SPAN>is the forward target — the tunnel listener on the loopback interface.</P> <P class="pw-post-body-paragraph abx aby xg abz b aca acc acd ace acg ach aci ack acl acm aco acp acq acs act afr acu vi bg" data-selectable-paragraph=""><STRONG class="abz pf">3.</STRONG><SPAN> </SPAN>Next, the health check. We implement it as a small Python HTTP server. On every incoming GET request, it attempts a TCP connection to the tunnel port on the loopback interface. If the connection succeeds, it returns<SPAN> </SPAN><EM class="adv">HTTP 200</EM>; otherwise,<SPAN> </SPAN><EM class="adv">503</EM>. A sample implementation:</P> <LI-CODE lang="python"># /usr/local/bin/tunnel-health-check.py #!/usr/bin/env python3 import socket import http.server TUNNEL_PORT = 13306 class HealthHandler(http.server.BaseHTTPRequestHandler): def do_GET(self): try: s = socket.create_connection(("127.0.0.1", TUNNEL_PORT), timeout=3) s.close() self.send_response(200) except Exception: self.send_response(503) self.end_headers() def log_message(self, *args): pass http.server.HTTPServer(("0.0.0.0", 8080), HealthHandler).serve_forever()</LI-CODE> <P class="pw-post-body-paragraph abx aby xg abz b aca acc acd ace acg ach aci ack acl acm aco acp acq acs act afr acu vi bg" data-selectable-paragraph="">Make it executable:</P> <LI-CODE lang="python">sudo chmod +x /usr/local/bin/tunnel-health-check.py</LI-CODE> <P class="pw-post-body-paragraph abx aby xg abz b aca acb acc acd ace acf acg ach aci acj ack acl acm acn aco acp acq acr acs act acu vi bg" data-selectable-paragraph="">A sample systemd unit for the health check:</P> <LI-CODE lang="python"># /etc/systemd/system/tunnel-health-check.service [Unit] Description=Tunnel health check HTTP endpoint After=network.target [Service] ExecStart=/usr/bin/python3 /usr/local/bin/tunnel-health-check.py Restart=always RestartSec=5 [Install] WantedBy=multi-user.target</LI-CODE> <P class="pw-post-body-paragraph abx aby xg abz b aca acb acc acd ace acf acg ach aci acj ack acl acm acn aco acp acq acr acs act acu vi bg" data-selectable-paragraph="">The standard<SPAN> </SPAN><EM class="adv">sshd</EM><SPAN> </SPAN>on the proxy VM accepts the reverse SSH connection from the on-premises tunnel host. We leave GatewayPorts to no as the default, which keeps the tunnel listener safely on the loopback interface, as explained earlier. No changes to<SPAN> </SPAN><EM class="adv">sshd_config</EM><SPAN> </SPAN>are needed for the reverse tunnel to work.</P> <P class="pw-post-body-paragraph abx aby xg abz b aca acb acc acd ace acf acg ach aci acj ack acl acm acn aco acp acq acr acs act acu vi bg" data-selectable-paragraph="">Finally, enable and start both services:</P> <LI-CODE lang="python">sudo systemctl daemon-reload sudo systemctl enable socat-db-proxy tunnel-health-check sudo systemctl start socat-db-proxy tunnel-health-check</LI-CODE> <BLOCKQUOTE class="ads adt adu"> <P class="abx aby adv abz b aca acb acc acd ace acf acg ach aci acj ack acl acm acn aco acp acq acr acs act acu vi bg" data-selectable-paragraph=""><STRONG class="abz pf">Security Note:</STRONG><SPAN> </SPAN>Apply NSG rules on the proxy VM to limit access as suggested below.</P> <P class="abx aby adv abz b aca acb acc acd ace acf acg ach aci acj ack acl acm acn aco acp acq acr acs act acu vi bg" data-selectable-paragraph="">— Inbound SSH (port 22): Restrict to the on-premises CIDR or specific tunnel host IPs.</P> <P class="abx aby adv abz b aca acb acc acd ace acf acg ach aci acj ack acl acm acn aco acp acq acr acs act acu vi bg" data-selectable-paragraph="">— Inbound database port (e.g., 3306): Restrict to the Databricks VNet CIDR (classic) and the PLS NAT subnet (serverless).</P> <P class="abx aby adv abz b aca acb acc acd ace acf acg ach aci acj ack acl acm acn aco acp acq acr acs act acu vi bg" data-selectable-paragraph="">— Inbound health probe port (e.g., 8080): Restrict to the Azure load balancer probe source 168.63.129.16/32.</P> </BLOCKQUOTE> <P class="pw-post-body-paragraph abx aby xg abz b aca acb acc acd ace acf acg ach aci acj ack acl acm acn aco acp acq acr acs act acu vi bg" data-selectable-paragraph="">The proxy VM is now ready to accept reverse SSH tunnels and serve traffic to Databricks through the SLB.</P> <H2 id="c90f" class="acv acw xg bb acx ns acy nt nu nv acz nw nx ada adb adc add ade adf adg adh adi adj adk adl adm bg" data-selectable-paragraph="">On-premises side — the tunnel host</H2> <P class="pw-post-body-paragraph abx aby xg abz b aca adn acc acd ace ado acg ach aci adp ack acl acm adq aco acp acq adr acs act acu vi bg" data-selectable-paragraph="">The on-premises tunnel host is a Linux host (physical or virtual) with network access to the database and outbound SSH (port 22) to the proxy VMs. Because the tunnel needs to stay up through network blips, proxy VM restarts, and other connection drops, we use autossh, a wrapper around ssh that automatically reconnects when the SSH session dies. On the tunnel host, we set up three pieces: autossh, an SSH key pair for authenticating to each proxy VM, and one systemd service per proxy VM so the tunnel comes up at boot and stays alive.</P> <P class="pw-post-body-paragraph abx aby xg abz b aca acb acc acd ace acf acg ach aci acj ack acl acm acn aco acp acq acr acs act acu vi bg" data-selectable-paragraph=""><STRONG class="abz pf">1.<SPAN> </SPAN></STRONG>Install<SPAN> </SPAN><EM class="adv">autossh</EM><SPAN> </SPAN>from the package repository:</P> <LI-CODE lang="python">sudo apt-get install -y autossh</LI-CODE> <P class="pw-post-body-paragraph abx aby xg abz b aca acb acc acd ace acf acg ach aci acj ack acl acm acn aco acp acq acr acs act acu vi bg" data-selectable-paragraph=""><STRONG class="abz pf">2.<SPAN> </SPAN></STRONG>Generate an SSH key pair for the tunnel and copy the public key to each proxy VM:</P> <LI-CODE lang="python">ssh-keygen -t rsa -b 4096 -f ~/.ssh/id_rsa_tunnel -N "" ssh-copy-id -i ~/.ssh/id_rsa_tunnel <user>@<proxy-vm-1-ip> ssh-copy-id -i ~/.ssh/id_rsa_tunnel <user>@<proxy-vm-2-ip></LI-CODE> <P class="pw-post-body-paragraph abx aby xg abz b aca acb acc acd ace acf acg ach aci acj ack acl acm acn aco acp acq acr acs act acu vi bg" data-selectable-paragraph=""><STRONG class="abz pf">3.</STRONG><SPAN> </SPAN>Wrap<SPAN> </SPAN><EM class="adv">autossh</EM><SPAN> </SPAN>in a systemd service so it starts at boot and restarts if it ever exits. Because each autossh process maintains exactly one outbound SSH connection, we create one systemd service per proxy VM. A sample unit for the first proxy VM:</P> <LI-CODE lang="python"># /etc/systemd/system/ssh-tunnel-proxy1.service [Unit] Description=SSH Reverse Tunnel to Proxy VM 1 After=network-online.target Wants=network-online.target [Service] User=<user> ExecStart=/usr/bin/autossh -M 0 -N \ -R 13306:<db-ip>:3306 \ -o ServerAliveInterval=30 \ -o ServerAliveCountMax=3 \ -o ExitOnForwardFailure=yes \ -o StrictHostKeyChecking=accept-new \ -i /home/<user>/.ssh/id_rsa_tunnel \ <user>@<proxy-vm-1-ip> Restart=always RestartSec=10 [Install] WantedBy=multi-user.target</LI-CODE> <P class="pw-post-body-paragraph abx aby xg abz b aca acb acc acd ace acf acg ach aci acj ack acl acm acn aco acp acq acr acs act acu vi bg" data-selectable-paragraph="">Let’s walk through the flags:</P> <UL class=""> <LI id="e9b9" class="abx aby xg abz b aca acb acc acd ace acf acg ach aci acj ack acl acm acn aco acp acq acr acs act acu adw adx ady bg" data-selectable-paragraph=""><STRONG class="abz pf"><EM class="adv">-M 0<SPAN> </SPAN></EM>—<SPAN> </SPAN></STRONG>disables autossh’s built-in monitoring port. By itself, that means autossh would never detect a dead connection. We get that detection from SSH’s own keepalive mechanism, which is why -M 0 is always paired with ServerAliveInterval and ServerAliveCountMax.</LI> <LI id="3b07" class="abx aby xg abz b aca adz acc acd ace aea acg ach aci aeb ack acl acm aec aco acp acq aed acs act acu adw adx ady bg" data-selectable-paragraph=""><STRONG class="abz pf"><EM class="adv">ServerAliveInterval=30</EM></STRONG><SPAN> </SPAN>— sends a keepalive every 30 seconds.</LI> <LI id="80dc" class="abx aby xg abz b aca adz acc acd ace aea acg ach aci aeb ack acl acm aec aco acp acq aed acs act acu adw adx ady bg" data-selectable-paragraph=""><STRONG class="abz pf"><EM class="adv">ServerAliveCountMax=3</EM></STRONG><SPAN> </SPAN>— means after three missed keepalives (about 90 seconds), SSH treats the connection as dead and exits. At that point, systemd restarts autossh, which establishes a fresh tunnel.</LI> <LI id="ada2" class="abx aby xg abz b aca adz acc acd ace aea acg ach aci aeb ack acl acm aec aco acp acq aed acs act acu adw adx ady bg" data-selectable-paragraph=""><STRONG class="abz pf"><EM class="adv">ExitOnForwardFailure=yes</EM></STRONG><SPAN> </SPAN>— exits SSH immediately if the remote port forward cannot be bound, which is useful when a proxy VM is briefly unreachable at connect time.</LI> <LI id="340d" class="abx aby xg abz b aca adz acc acd ace aea acg ach aci aeb ack acl acm aec aco acp acq aed acs act acu adw adx ady bg" data-selectable-paragraph=""><STRONG class="abz pf"><EM class="adv">-R 13306:<db-ip>:3306</EM></STRONG><SPAN> </SPAN>— is the reverse port forward — open a listener on localhost:13306 on the proxy VM and forward everything to the on-premises database at <db-ip>:3306, which the tunnel host reaches over the local network.</LI> </UL> <P class="pw-post-body-paragraph abx aby xg abz b aca acb acc acd ace acf acg ach aci acj ack acl acm acn aco acp acq acr acs act acu vi bg" data-selectable-paragraph="">Create a matching<SPAN> </SPAN><EM class="adv">ssh-tunnel-proxy2.service</EM><SPAN> </SPAN>for the second proxy VM (change the service name, description, and target IP). Then enable and start both:</P> <LI-CODE lang="python">sudo systemctl daemon-reload sudo systemctl enable ssh-tunnel-proxy1 ssh-tunnel-proxy2 sudo systemctl start ssh-tunnel-proxy1 ssh-tunnel-proxy2</LI-CODE> <P class="pw-post-body-paragraph abx aby xg abz b aca acb acc acd ace acf acg ach aci acj ack acl acm acn aco acp acq acr acs act acu vi bg" data-selectable-paragraph="">For a multi-database setup, extend each<SPAN> </SPAN><EM class="adv">autossh</EM><SPAN> </SPAN>command with additional -R forwards, mapping a different high-port tunnel port to each database:</P> <LI-CODE lang="python">autossh ... \ -R 13306:<mysql-ip>:3306 \ -R 15432:<postgres-ip>:5432 \ -R 11433:<sqlserver-ip>:1433</LI-CODE> <P class="pw-post-body-paragraph abx aby xg abz b aca acb acc acd ace acf acg ach aci acj ack acl acm acn aco acp acq acr acs act acu vi bg" data-selectable-paragraph="">Each proxy VM’s socat configuration then bridges the corresponding NIC port to each tunnel port.</P> <BLOCKQUOTE class="ads adt adu"> <P class="abx aby adv abz b aca acb acc acd ace acf acg ach aci acj ack acl acm acn aco acp acq acr acs act acu vi bg" data-selectable-paragraph=""><STRONG class="abz pf">Security Note:</STRONG><SPAN> </SPAN>Lock down the on-premises tunnel host as well.</P> <P class="abx aby adv abz b aca acb acc acd ace acf acg ach aci acj ack acl acm acn aco acp acq acr acs act acu vi bg" data-selectable-paragraph="">— Outbound port 22: Restrict the on-premises firewall to allow port 22 outbound only to the specific proxy VM IPs.</P> <P class="abx aby adv abz b aca acb acc acd ace acf acg ach aci acj ack acl acm acn aco acp acq acr acs act acu vi bg" data-selectable-paragraph="">— SSH key hardening: Restrict the tunnel SSH key to forwarding only. On each proxy VM, find the entry for the tunnel public key in `~/.ssh/authorized_keys` (the one added by `ssh-copy-id` earlier) and prefix it with `restrict,permitlisten="13306"` (add a `permitlisten` for each forwarded port). This blocks shell access, PTY allocation, and agent forwarding.</P> </BLOCKQUOTE> <P class="pw-post-body-paragraph abx aby xg abz b aca acb acc acd ace acf acg ach aci acj ack acl acm acn aco acp acq acr acs act acu vi bg" data-selectable-paragraph="">With both ends in place, the tunnel is live — the on-premises host maintains one<SPAN> </SPAN><EM class="adv">autossh</EM><SPAN> </SPAN>process per proxy VM, each proxy VM has the tunnel listener on loopback plus socat bridging to the NIC, and the SLB probes the health check to detect and route around any tunnel failures.</P> <P class="pw-post-body-paragraph abx aby xg abz b aca acb acc acd ace acf acg ach aci acj ack acl acm acn aco acp acq acr acs act acu vi bg" data-selectable-paragraph=""> </P> <H2 id="7346" class="acv acw xg bb acx ns acy nt nu nv acz nw nx ada adb adc add ade adf adg adh adi adj adk adl adm bg" data-selectable-paragraph="">Failover testing</H2> <P class="pw-post-body-paragraph abx aby xg abz b aca adn acc acd ace ado acg ach aci adp ack acl acm adq aco acp acq adr acs act acu vi bg" data-selectable-paragraph="">To validate the HA design end-to-end, we ran a set of controlled failure tests against a two-VM proxy hub. The setup mirrored the architecture described above — two proxy VMs behind an SLB, a Databricks foreign catalog for JDBC queries, and a Lakeflow Connect CDC pipeline for a long-lived connection test. The goal was to confirm that a tunnel failure is detected quickly, does not interrupt Databricks traffic, and recovers automatically. We also ran a few negative tests. As shown in the table below, the results of these tests validate the specific design choices we made, such as the HTTP health check and pointing the connection at the SLB for CDC.</P> <TABLE> <TBODY> <TR> <TD> <P><STRONG>#</STRONG></P> </TD> <TD> <P><STRONG>Test scenario</STRONG></P> </TD> <TD> <P><STRONG>Expected result</STRONG></P> </TD> <TD> <P><STRONG>Observed result</STRONG></P> </TD> </TR> <TR> <TD> <P><SPAN>1</SPAN></P> </TD> <TD> <P><SPAN>JDBC query through SLB, both VMs are healthy</SPAN></P> </TD> <TD> <P><SPAN>Query succeeds (baseline)</SPAN></P> </TD> <TD> <P><SPAN>Query succeeded</SPAN></P> </TD> </TR> <TR> <TD> <P><SPAN>2</SPAN></P> </TD> <TD> <P><SPAN>JDBC query, one tunnel stop</SPAN></P> <P><SPAN>ped, plain TCP probe (no HTTP health check)</SPAN></P> </TD> <TD> <P><SPAN>TCP probe sees socat as healthy; SLB keeps routing to dead path; queries fail intermittently</SPAN></P> </TD> <TD> <P><SPAN>As expected — SLB kept routing to the dead path, queries failed intermittently</SPAN></P> </TD> </TR> <TR> <TD> <P><SPAN>3</SPAN></P> </TD> <TD> <P><SPAN>JDBC query, one tunnel stopped, HTTP health check enabled</SPAN></P> </TD> <TD> <P><SPAN>SLB detects and fails over; queries continue to succeed</SPAN></P> </TD> <TD> <P><SPAN>SLB detected within a few seconds, routed traffic to the remaining VM, and queries continued to succeed</SPAN></P> </TD> </TR> <TR> <TD> <P><SPAN>4</SPAN></P> </TD> <TD> <P><SPAN>Foreign catalog query, classic compute over VNet peering</SPAN></P> </TD> <TD> <P><SPAN>Query succeeds (baseline)</SPAN></P> </TD> <TD> <P><SPAN>Query succeeded</SPAN></P> </TD> </TR> <TR> <TD> <P><SPAN>5</SPAN></P> </TD> <TD> <P><SPAN>Foreign catalog query, serverless compute over PE → PLS → SLB</SPAN></P> </TD> <TD> <P><SPAN>Query succeeds (baseline)</SPAN></P> </TD> <TD> <P><SPAN>Query succeeded</SPAN></P> </TD> </TR> <TR> <TD> <P><SPAN>6</SPAN></P> </TD> <TD> <P><SPAN>Lakeflow CDC pipeline, connection points at a specific proxy VM, that VM's tunnel stopped</SPAN></P> </TD> <TD> <P><SPAN>Gateway keeps retrying the dead VM; pipeline fails persistently until the connection is changed</SPAN></P> </TD> <TD> <P><SPAN>As expected — gateway kept retrying the same dead VM, pipeline failed persistently until we reconnected at the SLB</SPAN></P> </TD> </TR> <TR> <TD> <P><SPAN>7</SPAN></P> </TD> <TD> <P><SPAN>Lakeflow CDC pipeline, connection points at SLB, one tunnel stopped</SPAN></P> </TD> <TD> <P><SPAN>Gateway reconnects through SLB to a healthy VM; pipeline resumes without data loss</SPAN></P> </TD> <TD> <P><SPAN>Existing gateway connection broke, retry reconnected through SLB, pipeline resumed from the last binlog position, no data loss</SPAN></P> </TD> </TR> </TBODY> </TABLE> <P><SPAN>The most important takeaway from all these tests — point Databricks connections at the SLB frontend rather than a specific proxy VM. The SLB is what carries the failover through.</SPAN></P> <P> </P> <H1><SPAN>Conclusion</SPAN></H1> <P><SPAN>The reverse SSH tunnel proxy hub we presented in this post is one of many patterns to connect Azure Databricks classic and serverless compute to on-premises resources. We have previously published blogs and detailed reference architectures for several such connectivity patterns. What makes this model different is the direction of connection initiation — the on-premises host initiates the connection outbound (on-premises to cloud) to avoid relaxing firewall restrictions, and the return traffic flows back (cloud to on-premises) over the established path.</SPAN></P> <P><SPAN>If your on-premises security policy prohibits inbound connections from the cloud, and changing that policy isn't on the table, the reverse SSH tunnel proxy hub gives you a documented, validated path to reach your databases from both Azure Databricks classic and serverless compute. The architecture requires no inbound firewall changes on-premises, recovers automatically from tunnel failures, and supports multiple databases over a single hub.</SPAN></P> <P class="pw-post-body-paragraph abx aby xg abz b aca adn acc acd ace ado acg ach aci adp ack acl acm adq aco acp acq adr acs act acu vi bg" data-selectable-paragraph=""><SPAN>For the Databricks-side configuration, refer to the official Azure Databricks documentation on </SPAN><A href="https://learn.microsoft.com/en-us/azure/databricks/security/network/serverless-network-security/ssh-reverse-tunnel" target="_blank" rel="noopener"><SPAN>Connect to on-premises databases using an SSH reverse tunnel.</SPAN></A></P> Fri, 01 May 2026 20:38:44 GMT KiranAnand 2026-05-01T20:38:44Z Azure Databricks — Reverse SSH Tunnel for On-Premises Connectivity https://community.databricks.com/t5/technical-blog/azure-databricks-reverse-ssh-tunnel-for-on-premises-connectivity/ba-p/155969 <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="ReverseTunnel-ArchDiagram.png" style="width: 400px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/26574iB746558C68733DBA/image-size/medium?v=v2&px=400" role="button" title="ReverseTunnel-ArchDiagram.png" alt="ReverseTunnel-ArchDiagram.png" /></span></P> Fri, 01 May 2026 20:38:44 GMT https://community.databricks.com/t5/technical-blog/azure-databricks-reverse-ssh-tunnel-for-on-premises-connectivity/ba-p/155969 KiranAnand 2026-05-01T20:38:44Z