Failed to Merge Fields Error on Delta Live Tables

[Jake2]

I'm running into an issue during the "Setting up Tables" phase of our DLT pipelines where I'm told a particular field is unable to be merged due to incompatible datatypes. See this example:

org.apache.spark.sql.AnalysisException: Failed to merge fields 'FOO' and 'FOO'. Failed to merge incompatible data types ByteType and DecimalType(1,0)

This field only occurs once on this table, but there is one other table in this pipeline that use this field. However, they do not flow into each other, they do not have the same source tables, and none their downstream tables interact with each other in the DAG. They are totally separate. 

This only seems to happen on regular refreshes. Full refreshes run without issue.

I'm not sure why it seems to be trying to merge these fields when they don't interact with each other. Has anyone else come across this?

Thanks

. 

[Kaniz]

Hi @Jake2 , The error message you're seeing is related to trying to merge two incompatible data types, ByteType and DecimalType(1,0), in the 'FOO' field. This is indicated by the error message "Failed to merge fields 'FOO' and 'FOO'. Failed to merge incompatible data types ByteType and DecimalType(1,0)".

Even though the fields don't interact with each other, Spark is still trying to merge them because it's part of its schema inference process. This process happens during the read operation, where Spark tries to infer the schema of the data it's reading. If there are incompatible data types in the same field across different data partitions, this error can occur.You may need to explicitly define the schema for your data to avoid this issue. This can be done using the .schema() function in Spark, where you provide the schema that your data should conform to.

[Jake2]

Hey Kaniz, I appreciate the response. 

I'm doing a lot of different tables in this pipeline. If explicitly defining the schemas is out of the question due to time constraints, would it work to just split the offending tables off into two pipelines?