data ingestion from external system - auth via client certificate

cuhlmann — Thu, 24 Oct 2024 12:34:33 GMT

Hi Community,

we have the requirement to ingest data in azure databricks from external systems.

Our customer ask us to use Client Certificate as authentication method.

Requests - https://requests.readthedocs.io/en/latest/user/advanced/

Aiohttp - https://docs.aiohttp.org/en/stable/client_advanced.html

Solace Broker API - https://docs.solace.com/API/API-Developer-Guide-Python/Python-API-Messaging-Service.htm#Client

The certificate is stored in a azure key vault but all three methods required a local file for Client Certificate authentication.

can someone advice us how to achieve this on azure databricks?

Thanks Christian

Re: data ingestion from external system - auth via client certificate

filipniziol — Thu, 24 Oct 2024 17:10:19 GMT

Hi @cuhlmann ,

As I understand you need to ingest data into Azure Databricks from external systems, and your customer requires using client certificate authentication. The challenge is that the client certificate is stored in Azure Key Vault, but the libraries you're using (Requests, Aiohttp, Solace Broker API) require a local certificate file for client authentication.

Here is the possible solution:

Create a key vault-backed secret scope in Databricks
Access secrets in the notebook like this:

# Accessing secrets from Key Vault client_cert = dbutils.secrets.get(scope="your_scope", key="client_certificate") client_key = dbutils.secrets.get(scope="your_scope", key="client_key")

Out of the secrets create a certificate file and save to the location (Volume when using Unity Catalog, DBFS tmp without Unity Catalog)
Use the certificate file in your code

import requests url = "https://external-system.example.com/api/data" # Use the certificate file for client authentication response = requests.get(url, cert=cert_file_path) # Check the response print(response.status_code) print(response.text)

topic data ingestion from external system - auth via client certificate in Administration & Architecture

data ingestion from external system - auth via client certificate

Re: data ingestion from external system - auth via client certificate