"Databricks" - "PySpark" - Read "JSON" file - Azure Blob container - "APPEND BLOB"

[hare]

Hi All,

 We are getting JSON files in Azure blob container and its "Blob Type" is "Append Blob".

We are getting an error "

AnalysisException: Unable to infer schema for JSON. It must be specified manually.", when we try to read using below mentioned script.

df = spark.read.json(source_location,multiLine=True,pathGlobFilter='2022-05-18T02_50_01_914Z_student.json')

df.createOrReplaceTempView('v_df')

spark.sql("select count(*) from v_df").display()

can anyone please do let me know if we have any option to read JSON files which has the blob type "Append Blob"? - We are using "Databricks" - "PySpark"

[Kaniz]

Hi @Hareesh Makam​ , Please check the path and try again.

[Kaniz]

Hi @Hareesh Makam​ , We haven’t heard from you on the last response from me, and I was checking back to see if you have a resolution yet. If you have any solution, please share it with the community as it can be helpful to others. Otherwise, we will respond with more details and try to help.

[User16856839485]

There currently does not appear to be direct support for append blob reads, however, converting the append blob to block blob [and then parquet or delta, etc.] are a viable option:

https://kb.databricks.com/en_US/data-sources/wasb-check-blob-types?_ga=2.258782666.1514035379.166567...