Databricks Access Bundles - config data needed by notebook
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-26-2023 11:46 PM
I have this structure -
Folder-1 - the root of databricks access directory. "databricks.yaml" file is in this directory
Folder-1 / Folder-2 has notebooks. One of the notebook, "test-notebook" is used for *job* configuration in databricks.yaml file.
Folder-1 / Folder-2 / configs / config.json is used by "test-notebook". When running the notebook directly in Databricks UI, I can read the contents of config.json and use json data successfully. I use the path 'configs/config.json'
However, when I deploy bundle and run the job, I get FileNotFound error. From the context of a job, the file configs / config.json is not accessible.
What is the best way to save configuration files within an access bundle and how to ensure that the file is available to the notebook?
Thanks,
Archana
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-27-2023 04:01 AM
@GiggleByte adding few more inputs to what @Retired_mod mentioned, @GiggleByte how you are running job, using cmd/vs console/github/gitalab. your resource folder should have converted yaml of your json not json to run dab jobs from other than databricks UI if i am not wrong
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-27-2023 07:25 AM
I am running the job via "databricks bundle run" command via Azure DevOps pipeline.
I tried using "files" section to databricks.yaml file, but when the notebook runs, it is unable to find conf/config1.json
@Retired_mod - config files are part of the bundle.
@karthik_p - so, is it mandatory that all config information needed by notebooks be in yaml format? If yes, then how to notebooks access parameters? For example, if I have a config file called "custom_configs.yaml", then will the notebook be able to open the file "custom_configs.yaml"? Or, is there another way to read the config information?
Thanks,
Archana
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-28-2023 02:46 PM
@GiggleByte @Yes based on demo test that I have done, it is working as you said.
JSON converted yaml config for job setting need to be placed under resources, that yaml has job config setting, it looks similar to rest api json request converted in form of yaml.

