Using F Strings in filenames to read in files from mounted containers

jch — Thu, 29 Jun 2023 14:24:54 GMT

Below are 2 versions of codes I tried and the failure messages. Does anyone have advice how to form this properly?

I tried this

for x in range(2,num_organization_page,1): file_str = f"dbfs:/mnt/dlsi2023/raw/ds_pet_finder_organization{x}.json", print(file_str) organizations_df = spark.read.option("multiline","true").json(f"{file_str}")

and got this error:
('dbfs:/mnt/dlsi2023/raw/ds_pet_finder_organization2.json',)
IllegalArgumentException: java.net.URISyntaxException: Illegal character in scheme name at index 0: ('dbfs:/mnt/dlsi2023/raw/ds_pet_finder_organization2.json',)

I tried this

for x in range(2,num_organization_page,1): file_str = f"/mnt/dlsi2023/raw/ds_pet_finder_organization{x}.json", print(file_str) organizations_df = spark.read.option("multiline","true").json(f"{file_str}")

and got this error:

('/mnt/dlsi2023/raw/ds_pet_finder_organization2.json',)
IllegalArgumentException: Path must be absolute: ('/mnt/dlsi2023/raw/ds_pet_finder_organization2.json',)

Re: Using F Strings in filenames to read in files from mounted containers

erigaud — Tue, 04 Jul 2023 11:54:00 GMT

Your code works, you just have to remove the comma at the end of the second line

file_str = f"dbfs:/mnt/dlsi2023/raw/ds_pet_finder_organization{x}.json",

Should be

file_str = f"dbfs:/mnt/dlsi2023/raw/ds_pet_finder_organization{x}.json"

This is what is causing the error.

Also note that

f"{file_str}" is equivalent to just file_str : No f-string needed here !

topic Using F Strings in filenames to read in files from mounted containers in Get Started Discussions

Using F Strings in filenames to read in files from mounted containers

Re: Using F Strings in filenames to read in files from mounted containers