topic Re: how to pass secrets keys using a spark_python_task in Data Engineering

how to pass secrets keys using a spark_python_task

jeremy98 — Thu, 03 Apr 2025 07:52:11 GMT

Hello community,

I was searching a way to pass secrets to spark_python_task. Using a notebook file is easy, it's only to use dbutils.secrets.get(...) but how to do the same thing using a spark_python_task set using serveless compute?

Kind regards,

Re: how to pass secrets keys using a spark_python_task

jeremy98 — Thu, 03 Apr 2025 08:53:30 GMT

ping, it's important 😞

Re: how to pass secrets keys using a spark_python_task

Renu_ — Thu, 03 Apr 2025 10:12:20 GMT

Hi @jeremy98. To securely access secrets in a spark_python_task with serverless compute in Databricks... first create a secret scope and add secrets (refer to this article). Then, pass secrets by injecting them into environment variables in the job configuration and access them in your Python script using os.environ().

You can also check out this blog for more details.

Re: how to pass secrets keys using a spark_python_task

analytics_eng — Tue, 28 Oct 2025 15:44:45 GMT

@Renu_ but passing them as spark_env will not work with serverless I guess? See also the limitations on the docs Serverless compute limitations | Databricks on AWS