Databricks Community

kulasangar · ‎03-18-2025

I have a python project and within that I do have a yaml file. Currently i'm building the project using poetry and creating an asset bundle to deploy it in Databricks as a workflow job.

So when the workflow runs, I do have an __init__.py within my entry point directory, so there I'm trying to read the yaml file content and update it on the fly using Python. I tested the below code locally with a dummy file and it worked fine.

def update_secrets_in_yml():
    with open("path of the file", "r") as file:
        data = yaml.load(file)

    logger.info("Updating the values from secrets")
    data['app_id'] = app_id # i will already have the value for this from the spark env variables in the cluster

    with open("path of the file", "w") as file:
        yaml.dump(data, file)

For eg imagine if there's a test.yaml like this

appid: ""

env: dev

So I would want to directly update the value for appid from the secrets and populate this yaml file during the runtime in Databricks. When I run the job, I'm able to read the file and print the content, but I'm unable to write/update anything back onto the file and that's when I get the Permission Denied error. Attaching the screenshot of the error below

What am I missing here?

mark_ott · Wednesday

The main issue is that Databricks jobs typically run in environments where the file system may be read-only or restricted—especially for files packaged within the asset bundle or inside locations like /databricks/driver, /databricks/conda, or other system-specific locations. This limitation means that while you can read configuration files bundled with your code, writing back to them directly usually fails with a "Permission Denied" error.

Why Does This Happen?

Asset Bundles: When deploying packages with Poetry or other asset bundling solutions, the bundled files are often placed in locations where you do not have write permissions.
Cluster Environment: Databricks clusters mount files from your asset bundle in read-only locations to prevent accidental overwrites or security issues.

1. Write to DBFS or a User-Writable Location

Instead of writing back to the original YAML file packaged in your asset bundle, write the updated config to a location like /dbfs/tmp/yourfile.yaml or the current working directory if permitted. Example:

python

import yaml
import os

def update_secrets_in_yml(app_id, input_path, output_path):
    with open(input_path, "r") as file:
        data = yaml.safe_load(file)

    data['appid'] = app_id

    with open(output_path, "w") as file:
        yaml.safe_dump(data, file)

# Usage:
input_path = "/path/to/readable/test.yaml"
output_path = "/dbfs/tmp/test.yaml"  # or "./test.yaml" if allowed
update_secrets_in_yml(app_id, input_path, output_path)

2. Keep Source YAML as Read-Only, and Use a Copy

On workflow start, read original YAML, update it, and work from the new copy.
Do not attempt to overwrite the source YAML within the asset bundle.

3. Use Environment Variables Instead

If you only need the value at runtime, consider passing secrets/config as environment variables or using Databricks secrets directly, avoiding file modifications.

How to Check Write Permissions

Try os.access(path, os.W_OK) on your intended write location to verify permissions.
In notebooks, use %fs ls /path or %fs head /path/filename to inspect DBFS and other directories.

Why Your Local Testing Works

Locally, both reading and writing are permitted in your filesystem. But in Databricks, permissions and location restrictions apply to files inside the asset bundle, causing your code to fail only at write-time.

References

Solution/Advice	Details	Citation
Asset bundle locations are read-only	Can't edit directly once deployed
Write to DBFS or `/tmp/` directories	Use user-writable locations
Environment variable usage suggested	Avoid file modifications for secrets

You are encountering "Permission Denied" because bundled files in Databricks are almost always read-only at runtime—write your updated YAML to a user-writable location like DBFS, /tmp/, or your working directory instead