topic I met with an issue when I was trying to use autoloader to read json files from Azure ADLS Gen2. I am getting this issue for specific files only. I checked the file are good and not corrupted. in Data Engineering

I met with an issue when I was trying to use autoloader to read json files from Azure ADLS Gen2. I am getting this issue for specific files only. I checked the file are good and not corrupted.

SRK — Fri, 02 Dec 2022 01:10:41 GMT

I met with an issue when I was trying to use autoloader to read json files from Azure ADLS Gen2. I am getting this issue for specific files only. I checked the file are good and not corrupted.

Following is the issue:

java.lang.IllegalArgumentException: requirement failed: Literal must have a corresponding value to string, but class Integer found.

com.databricks.sql.io.FileReadException: Error while reading file /mnt/Source/kafka/customer_raw/filtered_data/year=2022/month=11/day=9/hour=15/part-00000-31413bcf-0a8f-480f-8d45-6970f4c4c9f7.c000.json.

Detailed error attaching as a file:

I am using Delta Live Pipeline. Here is the code:

@dlt.table(

name = tablename,

comment = "Create Bronze Table",

table_properties={

"quality": "bronze"

}

)

def Bronze_Table_Create():

return (

spark

.readStream

.schema(schemapath)

.format("cloudFiles")

.option("cloudFiles.format", "json")

.option("cloudFiles.schemaLocation", schemalocation)

.option("cloudFiles.inferColumnTypes", "false")

.option("cloudFiles.schemaEvolutionMode", "rescue")

.load(sourcelocation)

)

This is too urgent. Any help is highly appreciated.

Re: I met with an issue when I was trying to use autoloader to read json files from Azure ADLS Gen2. I am getting this issue for specific files only. I checked the file are good and not corrupted.

Geeta1 — Fri, 02 Dec 2022 05:17:54 GMT

Hey @Swapnil Kamle , can you try keeping inferColumnTypes to true, by default JSON should consider all columns as string, not sure why it is failing.

Re: I met with an issue when I was trying to use autoloader to read json files from Azure ADLS Gen2. I am getting this issue for specific files only. I checked the file are good and not corrupted.

SRK — Fri, 02 Dec 2022 05:33:12 GMT

I can't make InferColumnTypes to true, as i am passing the schema explicitly. i don't want to infer columns. It's failing for few files only. I checked the files as well. however the files looks good.

Re: I met with an issue when I was trying to use autoloader to read json files from Azure ADLS Gen2. I am getting this issue for specific files only. I checked the file are good and not corrupted.

SRK — Fri, 02 Dec 2022 09:34:56 GMT

I got the issue resolved. The issues was by mistake we have duplicate columns in the schema files. Because of that it was showing that error. However, the error is totally mis-leading, that's why didn't able to rectify it.