cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

I met with an issue when I was trying to use autoloader to read json files from Azure ADLS Gen2. I am getting this issue for specific files only. I checked the file are good and not corrupted.

SRK
Contributor III

I met with an issue when I was trying to use autoloader to read json files from Azure ADLS Gen2. I am getting this issue for specific files only. I checked the file are good and not corrupted.

Following is the issue:

java.lang.IllegalArgumentException: requirement failed: Literal must have a corresponding value to string, but class Integer found.

com.databricks.sql.io.FileReadException: Error while reading file /mnt/Source/kafka/customer_raw/filtered_data/year=2022/month=11/day=9/hour=15/part-00000-31413bcf-0a8f-480f-8d45-6970f4c4c9f7.c000.json.

Detailed error attaching as a file:

 I am using Delta Live Pipeline. Here is the code:

@dlt.table(

 name = tablename,

 comment = "Create Bronze Table",

 table_properties={

 "quality": "bronze"

 }

)

def Bronze_Table_Create():

 return (

 spark

 .readStream

 .schema(schemapath)

 .format("cloudFiles")

 .option("cloudFiles.format", "json")

 .option("cloudFiles.schemaLocation", schemalocation)

 .option("cloudFiles.inferColumnTypes", "false")

 .option("cloudFiles.schemaEvolutionMode", "rescue")  

 .load(sourcelocation)

 )

This is too urgent. Any help is highly appreciated.

1 ACCEPTED SOLUTION

Accepted Solutions

SRK
Contributor III

I got the issue resolved. The issues was by mistake we have duplicate columns in the schema files. Because of that it was showing that error. However, the error is totally mis-leading, that's why didn't able to rectify it.

View solution in original post

3 REPLIES 3

Geeta1
Valued Contributor

Hey @Swapnil Kamle​ , can you try keeping inferColumnTypes to true, by default JSON should consider all columns as string, not sure why it is failing.

SRK
Contributor III

I can't make InferColumnTypes to true, as i am passing the schema explicitly. i don't want to infer columns. It's failing for few files only. I checked the files as well. however the files looks good.

SRK
Contributor III

I got the issue resolved. The issues was by mistake we have duplicate columns in the schema files. Because of that it was showing that error. However, the error is totally mis-leading, that's why didn't able to rectify it.

Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!