NiFi on EKS Fails to Connect to Databricks via JDBC – "Connection reset" Error

[December]

I'm using Apache Nifi (running on AWS EKS) to connect to Databricks (with compute on EC2) via JDBC. My JDBC URL is as follows:

jdbc:databricks://server_hostname:443/default;transportMode=http;ssl=1;httpPath=my_httppath;AuthMech=3;UID=token;PWD=my_token. However, I encountered the following error:

I have verified that:

• I can successfully telnet from my NiFi pod to the Databricks host on port 443.
• The necessary network security settings (firewalls, security groups) allow connections from the NiFi IP range.
• I can connect to Databricks using the same JDBC URL from DBeaver on my pc without any issues.

Can anyone please explain why NiFi is encountering a "Connection reset" error when connecting to Databricks via JDBC, and provide suggestions on how to resolve this issue?

[mark_ott]

A "Connection reset" error in NiFi when connecting to Databricks via JDBC, despite successful telnet and working connectivity from DBeaver, usually points to subtle protocol or compatibility issues rather than network-level blocks.​

Common Causes

• JDBC Driver Version mismatch: NiFi and DBeaver may use different versions of the Databricks JDBC driver. DBeaver often auto-downloads the latest compatible driver, while NiFi depends on the specific JAR file in its lib directory. Incompatible or outdated drivers are a frequent cause of SSL or protocol "connection reset" errors.​

• Java Environment Differences: The Java version or SSL/TLS implementation used inside the NiFi pod may differ from your local environment. Some Java distributions or containerized setups lack certain root CAs, have different cipher preferences, or miss JVM options required for Databricks.​

• Databricks Endpoint Security: Databricks may require certain headers or TLS ciphers that NiFi’s Java runtime doesn’t negotiate by default, even if telnet works. Packet inspection or SSL inspection by internal firewalls can sometimes allow telnet but block actual JDBC SSL initiation.​

Troubleshooting Steps

• Verify JDBC JAR: Ensure the exact same Databricks JDBC driver JAR (version) used by DBeaver is also present in NiFi’s lib directory, and that there are no older conflicting JARs.

• Check Java Version and Certificates: Confirm the NiFi pod’s JVM is using a recent Java version (1.8 or newer, preferably 11 or 17) and that its truststore includes the required root CAs for Databricks.

• Increase Log Verbosity: Enable DEBUG logs for the NiFi JDBC processor and DBCPConnectionPool to inspect full handshake/error stack traces.

• Test SSL Connectivity: Use tools like openssl s_client -connect server_hostname:443 from the NiFi pod to validate SSL negotiation, CA trust, and verify there’s no deep packet/SSL inspection interfering.

• Update JVM Options: Some environments require customizing JVM flags, e.g., -Dcom.sun.security.enableAIAcaIssuers=true or -Djdk.tls.client.protocols=TLSv1.2.

• Firewall/Proxy Inspection: If using security appliances between EKS and Databricks, ensure they support pass-through and do not alter SSL handshake traffic.

Additional Tips

• Compare Stack Traces: Differences in error messages between NiFi and DBeaver provide clues (e.g., SSLException vs. SocketException).

• Review Databricks Docs: Double-check Databricks JDBC prerequisites; sometimes VPC endpoints or private link setups need extra configuration.​

Resolving this usually involves aligning JDBC and Java runtime configurations between NiFi and your successfully working DBeaver setup, and ensuring no intermediary network component alters SSL traffic.​