JWT Encoding error while using Azure secret key

[Wycliff]

My secret value in Azure key vault is like below.
private_key="""-----BEGIN RSA PRIVATE KEY-----
****
****
-----END RSA PRIVATE KEY-----"""

Running this command in Databricks notebook - jwt.encode(claim_set,private_key,algorithm='RS256')

While using the above hardcoded secret value in the command, it is working well.

But failing while using keyvaults in the command - private_key=dbutils.secrets.get(scope="scope_nm",key="key_nm").

Error : ('Could not deserialize key data.The data may be in an incorrect format, it may be encrypted with an unsupported algorithm, or it may be an unsupported key type(e.g. EC curves with explicit parameters).'

[Kaniz_Fatma]

Hi @Wycliff, It seems you’re encountering an issue while using Azure Key Vault secrets in your Databricks Notebook. 

Let’s troubleshoot this!

The error message you’re seeing, “Could not deserialize key data,” indicates that there might be an issue with the format or type of the secret key retrieved from the Key Vault. 

Let’s break down the steps to resolve this:

Secret Scopes in Databricks:

• Ensure that you’ve set up a secret scope in Databricks that corresponds to your Azure Key Vault. Secret scopes allow you to securely store and manage secrets (such as private keys) in Databricks.
• The secret scope name should be unique within your workspace and follow certain naming conventions (alphanumeric characters, dashes, underscores, @, and periods).
• If you haven’t created a secret scope yet, you can do so by following the documentation on Azure Databricks Secret Scopes.

Accessing Secrets:

• When retrieving a secret from the secret scope, use the following syntax:private_key = dbutils.secrets.get(scope="scope_nm", key="key_nm") Replace "scope_nm" with the actual name of your secret scope and "key_nm" with the specific key name for your private key.
• Make sure that the secret key you’re retrieving is indeed the private key in the correct format (RSA private key in your case).

Key Format:

• Verify that the private key stored in your Key Vault is in the correct format. RSA private keys are typically PEM-encoded and have a specific structure.
• Ensure that there are no extra characters (such as newline characters) in the key.
• If you suspect any issues with the key format, you might need to regenerate the key and store it again in the Key Vault.

Unsupported Algorithms or Key Types:

• The error message also mentions unsupported algorithms or key types (e.g., EC curves with explicit parameters). Make sure that the algorithm used for encoding (RS256 in your case) is supported by the key type.
• If you’re using a different algorithm, ensure that it aligns with the type of key you’re retrieving.

Debugging:

• To debug further, print the retrieved private_key and verify its content. You can use print(private_key) to check if it matches the expected RSA private key format.
• If needed, log additional information (such as the key type) to identify any discrepancies.

Permissions and Access:

• Confirm that the Databricks cluster has the necessary permissions to access the secret scope and retrieve secrets from the Key Vault.
• Check if the service principal or user assigned to the Databricks cluster has the appropriate permissions in Azure Key Vault.

Remember that using secret scopes allows you to avoid hardcoding sensitive information directly in your notebooks, which is a best practice for security. Once you’ve verified the above steps, your JWT encoding command should work seamlessly with the secret retrieved from the Key Vault.

If you encounter any further issues, feel free to ask for additional assistance! 😊

[Wycliff]

Thanks much for your troubleshooting methods.

Validated the secret scopes, accessing secrets. These looks fine.

Key format - I feel problem is with the key format only. As of now I'm awaiting on Azure subscription access. 

But I printed the secret value in Databricks compared the same with actual RSA private key in notepad++ and look for any differences in CR LF. Below is the command used for printing the secret value.

value=dbutils.secrets.get(scope="scope_nm", key="key_nm")

for char in value:

      print(char,end='\u200B')

Got the secret value in zero Width Space format. Having the actual RSA key value in CR LF format. Any way to get the secret value as is?

on permission end, we are good. Can be able to access other secrets.

Not sure about validating the algorithm. Regenerating the RSA key also seems to be impossible now as it is dependent on external team.

Please help to proceed from here. Thanks.

On Permission