Using F Strings in filenames to read in files from mounted containers
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-29-2023 07:24 AM
Below are 2 versions of codes I tried and the failure messages. Does anyone have advice how to form this properly?
I tried this
for x in range(2,num_organization_page,1):
file_str = f"dbfs:/mnt/dlsi2023/raw/ds_pet_finder_organization{x}.json",
print(file_str)
organizations_df = spark.read.option("multiline","true").json(f"{file_str}")
and got this error:
('dbfs:/mnt/dlsi2023/raw/ds_pet_finder_organization2.json',)
IllegalArgumentException: java.net.URISyntaxException: Illegal character in scheme name at index 0: ('dbfs:/mnt/dlsi2023/raw/ds_pet_finder_organization2.json',)
I tried this
for x in range(2,num_organization_page,1):
file_str = f"/mnt/dlsi2023/raw/ds_pet_finder_organization{x}.json",
print(file_str)
organizations_df = spark.read.option("multiline","true").json(f"{file_str}")
and got this error:
('/mnt/dlsi2023/raw/ds_pet_finder_organization2.json',)
IllegalArgumentException: Path must be absolute: ('/mnt/dlsi2023/raw/ds_pet_finder_organization2.json',)
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
07-04-2023 04:54 AM
Your code works, you just have to remove the comma at the end of the second line
file_str = f"dbfs:/mnt/dlsi2023/raw/ds_pet_finder_organization{x}.json",
Should be
file_str = f"dbfs:/mnt/dlsi2023/raw/ds_pet_finder_organization{x}.json"
This is what is causing the error.
Also note that

