12-01-2022 05:10 PM
I met with an issue when I was trying to use autoloader to read json files from Azure ADLS Gen2. I am getting this issue for specific files only. I checked the file are good and not corrupted.
Following is the issue:
java.lang.IllegalArgumentException: requirement failed: Literal must have a corresponding value to string, but class Integer found.
com.databricks.sql.io.FileReadException: Error while reading file /mnt/Source/kafka/customer_raw/filtered_data/year=2022/month=11/day=9/hour=15/part-00000-31413bcf-0a8f-480f-8d45-6970f4c4c9f7.c000.json.
Detailed error attaching as a file:
I am using Delta Live Pipeline. Here is the code:
@dlt.table(
name = tablename,
comment = "Create Bronze Table",
table_properties={
"quality": "bronze"
}
)
def Bronze_Table_Create():
return (
spark
.readStream
.schema(schemapath)
.format("cloudFiles")
.option("cloudFiles.format", "json")
.option("cloudFiles.schemaLocation", schemalocation)
.option("cloudFiles.inferColumnTypes", "false")
.option("cloudFiles.schemaEvolutionMode", "rescue")
.load(sourcelocation)
)
This is too urgent. Any help is highly appreciated.
12-02-2022 01:34 AM
I got the issue resolved. The issues was by mistake we have duplicate columns in the schema files. Because of that it was showing that error. However, the error is totally mis-leading, that's why didn't able to rectify it.
12-01-2022 09:17 PM
Hey @Swapnil Kamle , can you try keeping inferColumnTypes to true, by default JSON should consider all columns as string, not sure why it is failing.
12-01-2022 09:33 PM
I can't make InferColumnTypes to true, as i am passing the schema explicitly. i don't want to infer columns. It's failing for few files only. I checked the files as well. however the files looks good.
12-02-2022 01:34 AM
I got the issue resolved. The issues was by mistake we have duplicate columns in the schema files. Because of that it was showing that error. However, the error is totally mis-leading, that's why didn't able to rectify it.
Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections.
Click here to register and join today!
Engage in exciting technical discussions, join a group with your peers and meet our Featured Members.