I met with an issue when I was trying to use autoloader to read json files from Azure ADLS Gen2. I am getting this issue for specific files only. I checked the file are good and not corrupted.

Following is the issue:

java.lang.IllegalArgumentException: requirement failed: Literal must have a corresponding value to string, but class Integer found.

com.databricks.sql.io.FileReadException: Error while reading file /mnt/Source/kafka/customer_raw/filtered_data/year=2022/month=11/day=9/hour=15/part-00000-31413bcf-0a8f-480f-8d45-6970f4c4c9f7.c000.json.

Detailed error attaching as a file:

 I am using Delta Live Pipeline. Here is the code:

@dlt.table(

 name = tablename,

 comment = "Create Bronze Table",

 table_properties={

 "quality": "bronze"

 }

)

def Bronze_Table_Create():

 return (

 spark

 .readStream

 .schema(schemapath)

 .format("cloudFiles")

 .option("cloudFiles.format", "json")

 .option("cloudFiles.schemaLocation", schemalocation)

 .option("cloudFiles.inferColumnTypes", "false")

 .option("cloudFiles.schemaEvolutionMode", "rescue")  

 .load(sourcelocation)

 )

This is too urgent. Any help is highly appreciated.