Hello Databricks Community,
I am facing a critical issue where our cluster fails to start when using an init script designed to install the Databricks ODBC driver. I'm hoping someone can shed light on what might be happening within our specific environment.
The Goal:
My objective is to connect from a Python notebook on an all-purpose compute cluster in Workspace A to a remote Databricks SQL Warehouse in Workspace B.
- Source: Python notebook on an All-Purpose Cluster (Databricks on Azure).
- Target: Remote Databricks SQL Warehouse (adb-....azuredatabricks.net).
Summary of My Debugging Journey:
databricks-sql-connector Fails: My first attempt using the standard databricks-sql-connector library hangs indefinitely and times out, suggesting a network firewall is blocking the connection.
pyodbc Fails (Initially): I then switched to using the pyodbc library, as this is more robust in complex network environments. This failed with the error: [unixODBC][Driver Manager]Can't open lib 'Databricks' : file not found This indicated the driver manager tools or the driver itself were not correctly installed or registered on the cluster OS.
Init Script Causes Cluster Failure: To solve the driver issue, I created an all-purpose compute cluster and attached the following init script, which should install all necessary components:
#!/bin/bash set -e sudo apt-get update sudo apt-get install -y unixodbc unixodbc-dev sudo dpkg -i "/Workspace/Users/my.user@email.com/path/to/databricksodbc_amd64.deb" sudo apt-get install -f
When I start the cluster with this init script, the cluster startup fails completely. The UI shows the error: Init script failure: ... failed: Script exit status is non-zero.
The Critical Finding (The Main Problem):
When I inspect the logs for the failed init script, the stderr log file does not contain any shell errors (like "permission denied" or "command not found").
Instead, the stderr log contains only a full Java Thread Dump, starting with lines like: "LDBasedSafeFlagClient" id=17 state=WAITING... "process reaper" id=15 state=RUNNABLE...
This seems to indicate that the apt-get or dpkg commands are so incompatible with our environment that they are causing a fatal crash of the core Databricks JVM services on the cluster node.
My Specific Questions:
- Has anyone ever seen this behavior where an init script using apt-get or dpkg causes a hard JVM crash instead of just a shell error?
- Is this thread dump a known symptom of a security-hardened Databricks environment (e.g., using a custom unmodifiable OS image, or a security agent like SentinelOne/CrowdStrike) that actively terminates processes that try to modify the system?
- Given this evidence, is it correct to conclude that my environment is fundamentally "locked down," and that installing any system-level software via init scripts is impossible by design?
It feels like I've hit a wall where the platform's security is preventing the necessary setup for an outbound ODBC connection. Any insight would be hugely appreciated.
Thank you
