"Databricks" - "PySpark" - Read "JSON" file - Azure Blob container - "APPEND BLOB"
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
05-19-2022 05:40 AM
Hi All,
We are getting JSON files in Azure blob container and its "Blob Type" is "Append Blob".
We are getting an error "
AnalysisException: Unable to infer schema for JSON. It must be specified manually.", when we try to read using below mentioned script.
df = spark.read.json(source_location,multiLine=True,pathGlobFilter='2022-05-18T02_50_01_914Z_student.json')
df.createOrReplaceTempView('v_df')
spark.sql("select count(*) from v_df").display()
can anyone please do let me know if we have any option to read JSON files which has the blob type "Append Blob"? - We are using "Databricks" - "PySpark"
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-13-2022 09:25 AM
There currently does not appear to be direct support for append blob reads, however, converting the append blob to block blob [and then parquet or delta, etc.] are a viable option:
