Relative Path Reading Ambiguity in running nested run commands

[data4life]

Hello All,

I came across an unusual error while using the %run & dbutils.notebook.run() functionalities of the notebook in tandem and the particular scenarios are listed below -

I have below directory structure(simplified) where all 3 notebooks are located - main, NB1 & NB2. PFB the screenshots of all 3 notebooks for depicting the commands & error.

Test/
│── main
│── Folder1/
│ ├── NB1
│── Folder2/
│ ├── NB2

Scenario 1: In the main notebook, I'm running %run "./Folder1/NB1" and then NB1 notebook further executes %run "./Folder2/NB2". Here the run fails in the main notebook while calling NB1 with below error(available in screenshot as well), even though the current working directory of both the %run are same  -

Error: Notebook not found: Shared/Test/Folder1/Folder2/NB2.

If the working directory is same for both %run then why its searching under '/Folder1/Folder2/NB2' rather it should have looked into '/Test/Folder2/NB2' ?

Scenario 2: In the main notebook, I'm running %run "./Folder1/NB1" and but now NB1 notebook executes dbutils.notebook.run("./Folder2/NB2",0) and %run command cell is commented out. This time the main notebook completes successfully. Here also the current working directory is same for both.

So, why is the relative path working differently in both scenarios, in spite the first call is via %run, which essentially executes all the called notebook codes inline ?
Requesting all the contributors to opine if they are aware on this & to Databricks technical team as well to explain the same.

Thanks in advance!_/\_
#nestedruns

[eniwoke]

Hi @data4life 
For scenario 1, I tried running the same code, and I believe I found the issue. It's related to the %run command in NB1.

The line %run "./Folder2/NB2" assumes that Folder2 is inside Folder1, but that's not the case in your setup. So, when you run NB1 directly, it throws the same error you saw when running "main" — because NB1 can't locate Folder2 within its directory.

To fix this, try updating the line in NB1 to:
%run "../Folder2/NB2"
The .. tells it to go up one directory level, allowing it to correctly find NB2 in Folder2.

Let me know if that helps!

[data4life]

Hey @eniwoke ,
Thanks for replying and trying this out. 
I understand that giving %run "../Folder2/NB2" solves the issue, but the question is why is it so that I need to traverse one step back ?
As %run essentially executes the calling notebook codes inline from where it is being called, which in this case is the directory where main notebook is present. Similar to what you have pointed out that the directory for dbutils.notebook.run("./Folder2/NB2", 0) is same as that of main notebook, don't you think same should hold true for all the NB1 notebook codes, which can even comprise of another %run(as in my case) command ? 
PFB the screenshots(I updated the path to ensure a successful run) where you will see both the %run commands have same current working directory. So it should have recognized the path %run "./Folder2/NB2" as its executing directory is at the same level where Folder2 is present.

[eniwoke]

Now, for Scenario 2: when you use dbutils.notebook.run in NB1 and call NB1 from main.py, it looks like the working directory is considered to be the one where main.py is located.

So, when NB1 executes the line dbutils.notebook.run("./Folder2/NB2", 0), it's effectively asking: "Where is Folder2 relative to the current working directory?" Since the current working directory is that of main.py, it won't be able to resolve the path correctly if Folder2 isn’t located relative to that directory.

[jameshughes]

I'm going to run an experiment in my workspace and let you know if I see the same thing.  I'm not sure if I have seen this, but also not sure if my use of relative pathing previously had notebooks in different directories as you have listed.  Generally, my main notebook is reference all other notebooks that are in the same directory.  I will report back what I see.