topic Re: Compile all the scripts under the workspace folder in Data Engineering

Compile all the scripts under the workspace folder

thushar — Wed, 22 Dec 2021 11:55:12 GMT

In workspace one folder I have around 100+ pyspark scripts, all these scripts need to be compiled before running the main program. In order to compile all these files, we are using the %run magic command like %run ../prod/netSales. Since we have 100+ such files, we wrote 100+ magic commands like in a notebook to compile all 100+ files.

Question is, is there any way to compile all the files under one folder in ADB workspace instead of one by one? Are there any iterative methods are available to go through eah the file and compile it.

Re: Compile all the scripts under the workspace folder

Hubert-Dudek — Wed, 22 Dec 2021 14:58:14 GMT

Problem is that you can list all files in workspace only via API call and than you can run every one of them using:

dbutils.notebook.run()

This is the script to list files from workspace (probably you need to add some filterning):

import requests
ctx = dbutils.notebook.entry_point.getDbutils().notebook().getContext()
host_name = ctx.tags().get("browserHostName").get()
host_token = ctx.apiToken().get()
 
response = requests.post(
    f'https://{host_name}/api/2.0/workspace/list',
    headers={'Authorization': f'Bearer {host_token}'},
    data = {'path': '<your-path>'}
  ).json()

Re: Compile all the scripts under the workspace folder

thushar — Thu, 23 Dec 2021 07:01:28 GMT

When I tried the above code I got the below error.

{'error_code': 'ENDPOINT_NOT_FOUND', 'message': "No API found for 'POST /workspace/list'"}

Re: Compile all the scripts under the workspace folder

Hubert-Dudek — Thu, 23 Dec 2021 13:10:37 GMT

what distribution are you using (community, Azure)? so I will update that code as it is quite old

Re: Compile all the scripts under the workspace folder

thushar — Thu, 23 Dec 2021 13:19:00 GMT

I didn't get you correctly.

I am using 9.1 LTS run time, ADB premium service, DS3_V2 instance... is it will help?

Re: Compile all the scripts under the workspace folder

Hubert-Dudek — Thu, 23 Dec 2021 13:48:25 GMT

so Azure - just tested , had to change to get and data to json. Please manipulate with path to get data from folder which you need:

import requests
ctx = dbutils.notebook.entry_point.getDbutils().notebook().getContext()
host_name = ctx.tags().get("browserHostName").get()
host_token = ctx.apiToken().get()
 
response = requests.get(
    f'https://{host_name}/api/2.0/workspace/list',
    headers={'Authorization': f'Bearer {host_token}',
            'Accept': 'application/json'},
    json = {'path': '/'}
  )

Re: Compile all the scripts under the workspace folder

Hubert-Dudek — Fri, 24 Dec 2021 11:00:43 GMT

did it help?

Re: Compile all the scripts under the workspace folder

thushar — Fri, 21 Jan 2022 10:38:52 GMT

Sorry for the delay, that was not completely fixed

Modified the code which you shared, and now I could able to get the list of files under the folder in the workspace.

In order to compile (not to run) the files, is there any way there? Can we use magic command %run in some loop or something?

Re: Compile all the scripts under the workspace folder

Atanu — Sat, 12 Feb 2022 16:20:34 GMT

@Thushar R does https://docs.databricks.com/notebooks/notebook-workflows.html notebook workflow help you on this scenario ?