Hi @jorperort I know there is no direct task like bash_task for jobs that allows you to run bash scripts without using notebook cells %sh or Python's subprocess. Have you considered using Init scripts for your cluster while setting up the job?With th...
Now, for Scenario 2: when you use dbutils.notebook.run in NB1 and call NB1 from main.py, it looks like the working directory is considered to be the one where main.py is located.So, when NB1 executes the line dbutils.notebook.run("./Folder2/NB2", 0),...
Hi @data4life For scenario 1, I tried running the same code, and I believe I found the issue. It's related to the %run command in NB1.The line %run "./Folder2/NB2" assumes that Folder2 is inside Folder1, but that's not the case in your setup. So, whe...
Hi @jahanzebbehan, unfortunately, I don't believe there is any way to decrease the downsizing time for a cluster, as this mostly happens automatically depending on workload volume.Here are some helpful links on autoscaling:https://community.databrick...
Hi @ChinuLee, the task-level description appears to be hidden in the UI, both during manual creation and when viewing tasks after creation. However, it is visible when you choose the "View as Code" option, as shown below:So it is supported, just hidd...
