How to refer to repository directories in Workflows

avesel — Mon, 31 Jul 2023 09:33:14 GMT

I need to refer to a configuration file which resides in a separate directory than the script. The paths and execution withing a notebook/python files work fine. When script is scheduled and uses code repository the directory names look obfuscated

[Databricks runtime]
repo dir: /Workspace/Repos/<user>/Python
/config/config.yaml
/tests/test_script.py

[Workflow]
repo dir: Workspace/Repos/.internal/af8a942fc5_commits
/c4e7042bee838df9e01242cc09cc81524cc07f24/config.yaml
/b01e96980f85e154fc22e20c225275216fefde80/test_script.py

How do I refer from within test_script.py and read config.yaml?
Note that directory name changes with each run...

Re: How to refer to repository directories in Workflows

lauraxyz — Wed, 22 Jan 2025 18:42:52 GMT

can you try with relative path?

like you get your current path within test_script.py with command like:

cur_path = os.getcwd()

then get the path to config.yaml with a relative path like:

config_path = os.path.abspath(os.path.join(cur_path, f"../config/config.yaml"))

topic How to refer to repository directories in Workflows in Data Engineering

How to refer to repository directories in Workflows

Re: How to refer to repository directories in Workflows