Issue with Lakebridge transpile installation – SSL Certificate Error

[aravindan_tk]

Hi Team,

I am trying to use Lakebridge to test a small piece of code for conversion. The base installation of Lakebridge worked fine, but when I attempt to install transpile, I encounter SSL-related errors. I even tried to hardcode of the certificates still failing.

Environment & Versions Tried

• Python 3.13

   

  22:40:24 ERROR [src/databricks/labs/lakebridge.install-transpile] URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: Missing Authority Key Identifier (_ssl.c:1032)>

  • Python 3.10 & 3.11

     

    Could not fetch URL https://pypi.org/simple/pip/: There was a problem confirming the ssl certificate: HTTPSConnectionPool(host='pypi.org', port=443): Max retries exceeded with url: /simple/pip/ (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:997)'))) - skipping

    What I Tried

    • Tested with multiple Python versions (3.10, 3.11, 3.13)

    • Installation of base Lakebridge works fine

    • Error only occurs while installing transpile

      Has anyone faced a similar issue? Could you please guide me on how to resolve the SSL certificate verification error during transpile installation?

      Thanks in advance!

[mark_ott]

The error indicates that while installing Lakebridge's transpile component on Databricks with Python 3.13, SSL certificate verification fails due to a "Missing Authority Key Identifier" in the certificate chain. This is a result of stricter requirements on certificate fields in recent Python and OpenSSL versions.

Create/Update Certificate Bundle: Fetch the full certificate chain, ensuring it contains required Authority Key Identifiers. Use:

 openssl s_client -showcerts -connect <your-host>:443 </dev/null | sed -n -e '/-BEGIN/,/-END/ p' > fullchain.pem

Then in Python,

import os
os.environ["REQUESTS_CA_BUNDLE"] = "/path/to/fullchain.pem"

This allows Python's requests library to validate using the updated CA bundle.

• Regenerate Certificates: If using local/test certificates (e.g., generated by minica), use the latest tool version that is compliant with RFC 5280 and Python 3.13's requirements.

• Use certifi CA Bundle: Install and use the certifi Python package, which provides a trusted, up-to-date CA root store:  In Python, run: import certifi
  os.environ["REQUESTS_CA_BUNDLE"] = certifi.where()
  This can resolve issues if system CA stores are out-of-date.As a last resort for testing only, you may disable SSL verification in your installation script:

import ssl
ssl._create_default_https_context = ssl._create_unverified_context

But this should never be used in production as it reduces security.