Is SSL cert needed for Azure Databricks API calls?

[asrivas]

Hi, I'm using Azure Databricks and making API calls to the endpoints with verify=False Eg.

response = requests.get(
'https://%s/api/2.0/clusters/list' % (databricks_domain),
headers=request_headers,
verify=False
)

Security scanners are flagging the use of verify=False

Question:
Is it necessary to install or configure an SSL certificate on the client side for API communication with Azure Databricks? If not, what’s the best practice for handling SSL verification with requests in this case? Should I enable verify=True, just remove verify=False or provide a certificate bundle?

[BigRoux · Accepted solution]

In API communication with Azure Databricks, it is not necessary to manually install or configure an SSL certificate on the client side because Azure Databricks endpoints inherently use TLS (Transport Layer Security) for secure communication. To address the security scanner flagging the use of verify=False, here are the best practices:

1. Enable SSL Verification (verify=True😞

   • Setting verify=True ensures that Python's requests library verifies the SSL certificate presented by the server, which is important for securing HTTPS connections. This is the recommended approach.
2. Provide a Certificate Bundle:

   • If the default CA certificates provided by your system are insufficient (e.g., due to network configurations or specific enterprise policies), you can explicitly provide a trusted certificate bundle using the verify parameter. For example: python response = requests.get( 'https://%s/api/2.0/clusters/list' % (databricks_domain), headers=request_headers, verify='/path/to/certifi/cacert.pem' ) Ensure the certificate bundle used is reliable and up-to-date.
3. Avoid Using verify=False:

   • Disabling SSL verification (verify=False) bypasses the security validation of the server's certificate, which raises security concerns like exposure to MITM (Man-in-the-Middle) attacks. Therefore, this approach should be avoided.

   Hope this helps. Cheers, Lou.

By adhering to these practices, you ensure secure and compliant communication with Azure Databricks endpoints. If unusual SSL-related issues arise, consider determining if your organization’s IT policies or network configurations require additional adjustments to the certificate validation process.