data ingestion from external system - auth via client certificate

[cuhlmann]

Hi Community,

we have the requirement to ingest data in azure databricks from external systems.

Our customer ask us to use Client Certificate as authentication method.

Requests - https://requests.readthedocs.io/en/latest/user/advanced/

Aiohttp - https://docs.aiohttp.org/en/stable/client_advanced.html

Solace Broker API - https://docs.solace.com/API/API-Developer-Guide-Python/Python-API-Messaging-Service.htm#Client

The certificate is stored in a azure key vault but all three methods required a local file for Client Certificate authentication.

can someone advice us how to achieve this on azure databricks?

Thanks Christian

[filipniziol]

Hi @cuhlmann ,

As I understand you need to ingest data into Azure Databricks from external systems, and your customer requires using client certificate authentication. The challenge is that the client certificate is stored in Azure Key Vault, but the libraries you're using (Requests, Aiohttp, Solace Broker API) require a local certificate file for client authentication.

Here is the possible solution:

• Create a key vault-backed secret scope in Databricks
• Access secrets in the notebook like this: 

# Accessing secrets from Key Vault
client_cert = dbutils.secrets.get(scope="your_scope", key="client_certificate")
client_key = dbutils.secrets.get(scope="your_scope", key="client_key")
​

• Out of the secrets create a certificate file and save to the location (Volume when using Unity Catalog, DBFS tmp without Unity Catalog)
• Use the certificate file in your code

import requests

url = "https://external-system.example.com/api/data"

# Use the certificate file for client authentication
response = requests.get(url, cert=cert_file_path)

# Check the response
print(response.status_code)
print(response.text)
​