AutoLoader issue - java.lang.AssertionError

[ayesharahmat]

The below error I am encountering . I am using microbatch for autoloader. please help to rectify this issue

java.lang.AssertionError: assertion failed: Invalid batch: path#36188,modificationTime#36189,length#36190L,content#36191,PROVIDER#36192,LOCATION#36193,REQUEST#36194,YEAR#36195,MONTH#36196,DAY#36197,HOUR#36198,MINUTE#36199,SECOND#36200 != path#40036,modificationTime#40037,length#40038L,content#40039

[Priyanka_Biswas]

Hello​ @Ayesha Rahmatali​ Could you please let me know which DBR version you are using?

The below error can occur if new partition columns are being inferred from your files which cause the issue.

If thats the case, in order to resolve the issue, please provide all partition columns in your schema or provide a list of partition columns which you would like to extract values for by using: .option("cloudFiles.partitionColumns", "<comma-separated-list|empty-string>". AutoLoader infers the partition columns as empty. Use cloudFiles.partitionColumns to explicitly parse columns from the directory structure.

For more input, kindly refer to the below document.

Reference: https://docs.databricks.com/spark/latest/structured-streaming/auto-loader-schema.html

[ayesharahmat]

Hi Priyanka,

Thanks for your reply. There was no partitions added into my delta table. So am not sure what to mention in that partitionColumn parameters. Is there any other scenarios where we can expect invalid batch failure

[Anonymous]

@Ayesha Rahmatali​ :

The error message you provided suggests that there is an assertion failure due to invalid batch data in your AutoLoader implementation. The error specifically indicates that the schema of the incoming data is not matching with the expected schema.

Here are some possible reasons that can cause this issue and their corresponding solutions:

1. Schema mismatch: The schema of the data in the batch might be different from the schema defined for the AutoLoader. Please ensure that the schema of the data in the batch matches the schema defined for the AutoLoader.
2. Corrupted data: The data in the batch might be corrupted or have some missing values. Please check if there are any null or empty values in the data.
3. Memory issue: It is possible that the batch size is too large for the system to handle. Please try reducing the batch size and see if the issue persists.
4. Network latency: It is possible that there is a network latency issue causing the data to arrive in an unexpected format. Please ensure that the network connection is stable and reliable.
5. Code issue: There might be an issue with the code you have written. Please review the code and check if there are any logical errors that might be causing the issue.

I hope these suggestions help you to identify and fix the issue.