I have a python project and within that I do have a yaml file. Currently i'm building the project using poetry and creating an asset bundle to deploy it in Databricks as a workflow job.

So when the workflow runs, I do have an __init__.py within my entry point directory, so there I'm trying to read the yaml file content and update it on the fly using Python. I tested the below code locally with a dummy file and it worked fine.

def update_secrets_in_yml():
    with open("path of the file", "r") as file:
        data = yaml.load(file)

    logger.info("Updating the values from secrets")
    data['app_id'] = app_id # i will already have the value for this from the spark env variables in the cluster

    with open("path of the file", "w") as file:
        yaml.dump(data, file)

For eg imagine if there's a test.yaml like this

appid: ""

env: dev

So I would want to directly update the value for appid from the secrets and populate this yaml file during the runtime in Databricks. When I run the job, I'm able to read the file and print the content, but I'm unable to write/update anything back onto the file and that's when I get the Permission Denied error. Attaching the screenshot of the error below

What am I missing here?