I'm encountering a puzzling schema merge issue with my Delta Live Table. My setup involves several master tables on Databricks, and due to a schema change in the source database, one of my Delta Live Tables has a column (e.g., "reference_score") that was originally an integer and has now been changed to a float.

After running the pipeline, I receive errors like:

com.databricks.pipelines.common.errors.DLTAnalysisException: [CANNOT_UPDATE_TABLE_SCHEMA] Failed to merge the current and new schemas for table ... To proceed with this schema change, you can trigger a full refresh of this table. Depending on your use case and the schema changes, you may be able to obviate the schema change -- you can update your queries so the output schema is compatible with the existing schema (e.g, by explicitly casting columns to the correct data type).
com.databricks.sql.transaction.tahoe.DeltaAnalysisException: [DELTA_FAILED_TO_MERGE_FIELDS] Failed to merge fields 'reference_score' and 'reference_score'
com.databricks.sql.transaction.tahoe.DeltaAnalysisException: [DELTA_MERGE_INCOMPATIBLE_DATATYPE] Failed to merge incompatible data types IntegerType and FloatType

Has anyone faced this issue? Is there a way to overwrite or merge the schema changes without triggering a full refresh? I'm curious if there are any workarounds or best practices to handle such a transformation scenario.

Thanks in advance!