topic Re: Incorrect secret value when loaded as environment variable in Data Engineering

Incorrect secret value when loaded as environment variable

ksilva — Fri, 29 Apr 2022 12:04:45 GMT

I recently faced an issue that took good hours to identify.

I'm loading an environment variable with a secret

ENVVAR: {{secrets/scope/key}}

The secret is loaded in my application, I could verify it's there, but its value is not correct. I realised that after comparing the environment variable value with the value of the secret when loading using dbutils

envvar = dbutils.secrets.get(scope="scope", key="key")

After a lot of trial and error, I realised that the problem is because my secret value has a dollar sign in it, and when loading through env vars the dollar sign and everything after it disappears, I suspect that it's somehow evaluated as an environment variable and gets replaced as well.

Example:

print(dbutils.secrets.get(scope="scope", key="key"))
# this would be "mysecret$here"
print(os.environ["ENVVAR"])
# this would be "mysecret"

Is it a bug on Databricks or is it something else?

Re: Incorrect secret value when loaded as environment variable

User16752242622 — Thu, 23 Jun 2022 18:40:51 GMT

Hi @kleber silva

There was a known issue which has been resolved now. That is when a $ character is included in a secret value, the $ and all subsequent text are truncated.

Although your question is actually related to how spark parse the value as an env variable, the fix for this is still in progress. We don't have an ETA for this yet.

Thank you for reporting this to us.

Re: Incorrect secret value when loaded as environment variable

jose_gonzalez — Mon, 25 Jul 2022 21:56:39 GMT

Hi @kleber silva,

Just a friendly follow-up. Did @Akash Bhat response help you to resolve your question? if it did, please mark it as best. Otherwise, please let us know if you still need help.

Re: Incorrect secret value when loaded as environment variable

danmlopsmaz — Wed, 05 Jun 2024 18:59:23 GMT

Hi team, is there an update or fix for this?

Re: Incorrect secret value when loaded as environment variable

Olaoye_Somide — Fri, 16 Aug 2024 09:21:17 GMT

Hi, what do you mean by a known issue? Can you point me to a documentation that has the fix? Thanks