Compile all the scripts under the workspace folder

[thushar]

In workspace one folder I have around 100+ pyspark scripts, all these scripts need to be compiled before running the main program. In order to compile all these files, we are using the %run magic command like %run ../prod/netSales. Since we have 100+ such files, we wrote 100+ magic commands like in a notebook to compile all 100+ files.

Question is, is there any way to compile all the files under one folder in ADB workspace instead of one by one? Are there any iterative methods are available to go through eah the file and compile it.

[Kaniz]

Hi @ thushar! My name is Kaniz, and I'm the technical moderator here. Great to meet you, and thanks for your question! Let's see if your peers in the community have an answer to your question first. Or else I will get back to you soon. Thanks.

[Hubert-Dudek · Accepted solution]

Problem is that you can list all files in workspace only via API call and than you can run every one of them using:

dbutils.notebook.run()

This is the script to list files from workspace (probably you need to add some filterning):

import requests
ctx = dbutils.notebook.entry_point.getDbutils().notebook().getContext()
host_name = ctx.tags().get("browserHostName").get()
host_token = ctx.apiToken().get()
 
response = requests.post(
    f'https://{host_name}/api/2.0/workspace/list',
    headers={'Authorization': f'Bearer {host_token}'},
    data = {'path': '<your-path>'}
  ).json()

[thushar]

When I tried the above code I got the below error.

{'error_code': 'ENDPOINT_NOT_FOUND', 'message': "No API found for 'POST /workspace/list'"}

[Hubert-Dudek]

what distribution are you using (community, Azure)? so I will update that code as it is quite old

[thushar]

I didn't get you correctly.

I am using 9.1 LTS run time, ADB premium service, DS3_V2 instance... is it will help?

[Hubert-Dudek]

so Azure - just tested , had to change to get and data to json. Please manipulate with path to get data from folder which you need:

import requests
ctx = dbutils.notebook.entry_point.getDbutils().notebook().getContext()
host_name = ctx.tags().get("browserHostName").get()
host_token = ctx.apiToken().get()
 
response = requests.get(
    f'https://{host_name}/api/2.0/workspace/list',
    headers={'Authorization': f'Bearer {host_token}',
            'Accept': 'application/json'},
    json = {'path': '/'}
  )

[Hubert-Dudek]

did it help?

[thushar]

Sorry for the delay, that was not completely fixed

Modified the code which you shared, and now I could able to get the list of files under the folder in the workspace.

In order to compile (not to run) the files, is there any way there? Can we use magic command %run in some loop or something?

[Atanu]

@Thushar R​  does https://docs.databricks.com/notebooks/notebook-workflows.html notebook workflow help you on this scenario ?