Re: Permission Denied while trying to update a yam...

mark_ott · ‎11-05-2025

The main issue is that Databricks jobs typically run in environments where the file system may be read-only or restricted—especially for files packaged within the asset bundle or inside locations like /databricks/driver, /databricks/conda, or other system-specific locations. This limitation means that while you can read configuration files bundled with your code, writing back to them directly usually fails with a "Permission Denied" error.

Why Does This Happen?

Asset Bundles: When deploying packages with Poetry or other asset bundling solutions, the bundled files are often placed in locations where you do not have write permissions.
Cluster Environment: Databricks clusters mount files from your asset bundle in read-only locations to prevent accidental overwrites or security issues.

1. Write to DBFS or a User-Writable Location

Instead of writing back to the original YAML file packaged in your asset bundle, write the updated config to a location like /dbfs/tmp/yourfile.yaml or the current working directory if permitted. Example:

python

import yaml
import os

def update_secrets_in_yml(app_id, input_path, output_path):
    with open(input_path, "r") as file:
        data = yaml.safe_load(file)

    data['appid'] = app_id

    with open(output_path, "w") as file:
        yaml.safe_dump(data, file)

# Usage:
input_path = "/path/to/readable/test.yaml"
output_path = "/dbfs/tmp/test.yaml"  # or "./test.yaml" if allowed
update_secrets_in_yml(app_id, input_path, output_path)

2. Keep Source YAML as Read-Only, and Use a Copy

On workflow start, read original YAML, update it, and work from the new copy.
Do not attempt to overwrite the source YAML within the asset bundle.

3. Use Environment Variables Instead

If you only need the value at runtime, consider passing secrets/config as environment variables or using Databricks secrets directly, avoiding file modifications.

How to Check Write Permissions

Try os.access(path, os.W_OK) on your intended write location to verify permissions.
In notebooks, use %fs ls /path or %fs head /path/filename to inspect DBFS and other directories.

Why Your Local Testing Works

Locally, both reading and writing are permitted in your filesystem. But in Databricks, permissions and location restrictions apply to files inside the asset bundle, causing your code to fail only at write-time.

References

Solution/Advice	Details	Citation
Asset bundle locations are read-only	Can't edit directly once deployed
Write to DBFS or `/tmp/` directories	Use user-writable locations
Environment variable usage suggested	Avoid file modifications for secrets

You are encountering "Permission Denied" because bundled files in Databricks are almost always read-only at runtime—write your updated YAML to a user-writable location like DBFS, /tmp/, or your working directory instead