How to copy notebooks from local to the tarrget folder via asset bundles

[BobCat62]

Hi all,

I am able to deploy Databricks assets to the target workspace. Jobs and workflows can also be created successfully.

But I have aspecial requirement, that I copy the note books to the target folder on databricks workspace.

Example:

on Local I have such a structure:

databricks/
├─ fixtures/
├─ src/
│ ├─ SubFolder1/
│ │ ├─ Notebook1.ipynb
│ ├─ SubFolder2/
│ │ ├─ Notebook2.ipynb
│ ├─ Notebook3.ipnyb
├─ resources/
│ ├─ dab.job.yml.css
├─ .gitignore

On Databricks side, I would like to have this:

Workspace/
├─ Repos/
├─ Shared/
├─ Users/
├─ SubFolder1/
│ ├─ Notebook1.ipynb
├─ Subfolder2/
│ ├─ Notebook2.ipynb
├─ Notebook3.ipnyb

How can I set DAB that I can copy the notebooks to the desired folder on workspace and not to the root path with other files

[ashraf1395]

Hi @BobCat62, we meet again 😃,

I hope you are doing great.

You can deploy your notebooks to your workspace, which are even outside your databricks.yml (bundle root path) using the sync paths mapping. Though by default all these resources go to your specified workspace root path. 

If you want to send these files to any target root path, you can do it using the following ways

#databricks.yml

targets:
  test:
    workspace:
      file_path: /Workspace/
    sync:
      path:
        - .src/subfolder1/*.ipynb
        - .src/subfolder2/*.ipynb
        - ./src/notebook3.ipynb

- I have placed all the sync and workspace mappings inside a target mapping you can have it at root level as well.
- I specified all the notebooks separately but in your case since all are in the src directory you can directly just put the ./src/* as well

- I have assumed that databricks.yml is in the root directory , if not you can change the path with relative to that.

Here are the databricks docs : 
- https://docs.databricks.com/aws/en/dev-tools/bundles/settings#file_path
- https://docs.databricks.com/aws/en/dev-tools/bundles/settings#paths

[BobCat62]

Hello @ashraf1395 ,

Nice to hear you and thank you for your hints.

Actually with your idea, I could reach half of my aim 😊

you can see here the folder structure in my VS code:

and here is part of my `databrick.yml` file:

targets:

  dev:

    # The default target uses 'mode: development' to create a development copy.

    # - Deployed resources get prefixed with '[dev my_user_name]'

    # - Any job schedules and triggers are paused by default.

    # See also https://docs.databricks.com/dev-tools/bundles/deployment-modes.html.

    mode: production

    #default: true

    workspace:

      host: https://adb-xxxxx.x.azuredatabricks.net

      root_path: /Workspace/Shared/.bundle/${bundle.name}/${bundle.target}

      file_path: /Workspace/

    sync:

      paths:

        - .src/emob1/*.ipynb

      exclude:

        - databricks/fixtures

        - databricks/resources

        - databricks/fixtures

        - databricks/.vscode  

        - databricks/src/.gitignor

        - databricks/src/README.md

If I deploy with this yml, I get this error:

Error: stat .src/emob1/*.ipynb: no such file or directory

I I remove this part:

 paths:

        - .src/emob1/*.ipynb

I can deploy successfully, but the result is not I want:

 I would like to have emob1 and emob2 directly under Workspace, and not under Databricks/src

Do you have any Idea?

Thanks