Failed to Merge Fields Error on Delta Live Tables

[Jake2]

I'm running into an issue during the "Setting up Tables" phase of our DLT pipelines where I'm told a particular field is unable to be merged due to incompatible datatypes. See this example:

org.apache.spark.sql.AnalysisException: Failed to merge fields 'FOO' and 'FOO'. Failed to merge incompatible data types ByteType and DecimalType(1,0)

This field only occurs once on this table, but there is one other table in this pipeline that use this field. However, they do not flow into each other, they do not have the same source tables, and none their downstream tables interact with each other in the DAG. They are totally separate. 

This only seems to happen on regular refreshes. Full refreshes run without issue.

I'm not sure why it seems to be trying to merge these fields when they don't interact with each other. Has anyone else come across this?

Thanks

. 

[Jake2]

Hey Kaniz, I appreciate the response. 

I'm doing a lot of different tables in this pipeline. If explicitly defining the schemas is out of the question due to time constraints, would it work to just split the offending tables off into two pipelines?

[PeterNRNLI]

This "solution" completely defeats the propose of using the mergeSchema option!
How can we automate our loads if we have to manually edit the schema all the time?
Isn't the point of mergeSchema was that we didn't need to manually define every column?