Issue Using Private CA Certificates for Databricks Serverless Private Git → On-Prem GitLab Connectio

kfadratek
New Contributor

‎11-07-2025 04:36 AM

Hi everyone,

I’m trying to properly configure Databricks Serverless Private Git to connect to our on-premises GitLab, but I'm running into issues with private CA certificates.

Following the latest Databricks recommendations, our connection to GitLab goes through:

  • a load balancer
  • a private proxy server
  • an on-prem GitLab instance
  • no public internet access (Databricks traffic cannot go out to the public Internet)

Because of this setup, we’re using Databricks Serverless Private Git.

What works

The connection to GitLab works correctly as long as SSL verification is disabled, for example:

{
"default": {
"sslVerify": false
}
}

With SSL verification turned off, Databricks can successfully clone the repository and run all Git operations.

What does NOT work

The issue appears when we want to enable proper SSL verification and use our internal CA certificate, since our organization does not use publicly-signed certificates — everything is signed by our internal PKI.

According to Databricks documentation:

  • we created the .git_settings folder in the workspace
  • we placed our CA certificate inside (e.g. ca.pem)
  • we added config.json to point Git to that CA file

Our configuration looks like this:

{
"default": {
"caCertPath": "/Workspace/.git_settings/cert.pem",
"httpProxy": "https://gitlab.apps.correcturl/"
},
"remotes": [
{
"caCertPath": "/Workspace/.git_settings/cert.pem",
"urlPrefix": "https://gitlab.apps.correcturl/"
}
]
}

After enabling this, the connection stops working. Git on Databricks fails to validate the certificate even though the CA is valid and works with other tools (curl/git from local machines, CI pipelines, etc.).

Symptoms

  • the connection fails with an SSL certificate verification error
  • it inside Databricks does not seem to see or load the custom CA certificate
  • disabling sslVerify makes everything work, but this is not acceptable from a security standpoint

Error creating Git folder
remote: git proxy error, HTTPSConnectionPool(host='gitlab.apps.correcturl', port=443): Max retries exceeded with url: /path-in-git/notebooks.git/info/refs?service=git-upload-pack (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get issuer certificate (_ssl.c:1007)')))
Request ID: 7faa538c-946e-416c-90ee-b41c5891ff85. Show error details

Question
Has anyone successfully implemented a similar setup:

  • Databricks → Serverless Private Git
  • On-prem GitLab (no public access)
  • Internal/private CA (enterprise PKI)

and managed to make Git use a custom CA certificate correctly?

Any ideas or guidance would be very appreciated.

Thanks!

Labels:
0 Kudos

Louis_Frolio
Databricks Employee
Databricks Employee

‎11-08-2025 01:55 PM

Hello @kfadratek ,  thanks for the detailed context — Let's take a look at what could be causing the SSL verification to fail with a custome CA in Serverless Private Git and discuss some approaches that might resolve it.

What’s likely going wrong

 
Based on the error “unable to get issuer certificate,” the most common causes in this setup are: * The CA bundle provided to Databricks doesn’t include the correct issuing chain (for example, only the root or only an intermediate instead of the full chain that issued the GitLab or proxy certificate).
  • The CA file or config.json isn’t readable to the serverless service (workspace ACLs not granting View) or the path is wrong. Serverless Private Git reads config from /Workspace/.git_settings/config.json, and you must grant all Git users View to that file and any referenced CA files. Changes can take up to 1 minute to apply.
  • The httpProxy value is mis-specified. In Serverless Private Git, httpProxy is the HTTPS URL of the proxy to route Git traffic through. It should point to your proxy (host:port), not to the Git server URL itself. The field is optional — leave it empty if you don’t need a proxy beyond your PrivateLink/NLB/SLB path.
  • The certificate presented by the load balancer/proxy doesn’t match the URL you’re using (CN/SAN mismatch) or is re-terminating TLS with a different issuer than you bundled.
  • You’re expecting logs from serverless, but logs aren’t currently available, so misconfigurations manifest only as connection errors. Troubleshooting must rely on config correctness and network validation.

Quick checks to run

  • Confirm the file paths exist exactly as referenced: * /Workspace/.git_settings/config.json * /Workspace/.git_settings/cert.pem (or your chosen name)
  • Set workspace permissions:
    • Grant View to all Git users for both the config file and CA file.
  • Verify the CA bundle content:
    • Concatenate the full chain that issued the certificate your serverless compute sees (root + all intermediates) into a single PEM file. Ensure the first cert in the file is the issuing CA for the server/proxy cert, and that the file contains only CA certs (not the server cert).
    • Test the bundle outside Databricks (curl with --cacert) against the exact hostname used by Serverless, then use that same bundle inside Databricks.
  • Validate hostname and chain:
    • Browse to https://gitlab.apps.correcturl/ from a machine that trusts only your enterprise CAs using your CA bundle; check the chain and SANs. If your LB/proxy offloads TLS, verify it presents the cert for gitlab.apps.correcturl and that the issuer matches the CA bundle.
  • Review your httpProxy configuration:
    • If you truly route Databricks traffic via an internal proxy, set httpProxy to that proxy’s full HTTPS URL (for example, https://proxy.internal.company.com:8443). Otherwise remove it; it’s optional.
  • Confirm NCC/private endpoint rules include the Git server FQDN (and any proxy FQDN if used) so Serverless can reach it through PrivateLink. Wait a few minutes after rules are created, then try a Git operation from the workspace.

Known requirements and structure to follow

  • Create /Workspace/.git_settings/config.json and keep the default section present (even partial). Use caCertPath to reference the custom CA and, if needed, httpProxy to route traffic. Remote-specific overrides go under remotes and must include urlPrefix.
  • Example minimal config (with proper proxy and CA bundle): json { "default": { "sslVerify": true, "caCertPath": "/Workspace/.git_settings/ca-bundle.pem", "httpProxy": "https://proxy.internal.company.com:8443" }, "remotes": [ { "urlPrefix": "https://gitlab.apps.correcturl/", "caCertPath": "/Workspace/.git_settings/ca-bundle.pem" } ] }
  • After saving, wait up to 1 minute for changes to take effect and then perform a Git operation (e.g., clone) to validate.

If you need deeper triage

 
Because serverless proxy logs aren’t available today, one reliable way to isolate CA/chain issues is to do a quick A/B test with the classic Git proxy on a small cluster and set the CA via environment variable: * Set GIT_PROXY_CA_CERT_PATH=/FileStore/myCA-bundle.pem on the proxy cluster. * Use the same bundle you intend to use for Serverless Private Git and validate clone/pull. If the classic proxy succeeds with your bundle but serverless fails, focus on workspace file access/permissions and the httpProxy semantics in the serverless config.
 

Summary of targeted fixes

  • Use a single CA bundle PEM with the full chain that issued the cert your LB/proxy presents.
  • Ensure workspace ACLs grant View to /Workspace/.git_settings/config.json and the CA file; confirm the paths are correct.
  • Set httpProxy to your actual proxy (host:port) if used; otherwise remove it.
  • Confirm NCC/private endpoint rules include the FQDNs you’re accessing and allow the path through PrivateLink.
  • Allow 1 minute for config changes to apply, then re-test clone/pull.
 
Hope this guidance gets you on the right track.
Cheers, Louis.

View solution in original post

0 Kudos