Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-04-2025 02:09 AM
Hello Guys,
I have some issue when i load several json files which have a same schema on databricks. when i do
2025_07_17_19_55_00_2025_07_31_21_55_00_17Q51D_alice_out.json 516.13 KB
2025_07_17_19_55_00_2025_07_31_21_55_00_17Q51D_bob_out.json 516.13 KB
2025_08_10_21_55_00_2025_08_24_21_55_00_17Q1D_alice_out.json 514.13 KB
2025_08_10_21_55_00_2025_08_24_21_55_00_17Q51D_bob_out.json 418.13 KB
options = {
"multiLine": True,
"inferSchema": True,
"allowUnquotedFieldNames": True,
"allowSingleQuotes": True,
"allowBackslashEscapingAnyCharacter": True,
"recursiveFileLookup": True,
}
df = spark.read.format("json").options(**options).load("Volumes/folder/dir1")
it pick up randomly two files
someone know how to solve this issue?
Cordially,
