topic MergeSchema Not Working in Data Engineering

MergeSchema Not Working

DJey — Fri, 02 Jun 2023 13:52:05 GMT

Hi All,

I have a scenario where my Exisiting Delta Table looks like below:

Now I have an incremental data with an additional column i.e. owner:

Dataframe Name --> scdDFBelow is the code snippet to merge Incremental Dataframe to targetTable, but the new column is not getting added:

spark.conf.set("spark.databricks.delta.schema.autoMerge.enabled",True) --> this is also enabled.

But still no luck. Below is the final result which i'm currently getting:

Data looks correct, but the only issue is New Column i.e. Owner is still not merged in targetTable.

Someone please help.

Thanks!

Re: MergeSchema Not Working

etsyal1e2r3 — Sun, 04 Jun 2023 00:49:50 GMT

Just add that column to the table with an ALTER TABLE statement. You should capture each table/dataframe columns as a list with df.columns, compare them, and if the table is missing anything do the alter table. Then run your code.

Re: MergeSchema Not Working

Anonymous — Tue, 06 Jun 2023 07:02:39 GMT

Hi @Divyansh Jain

Thank you for posting your question in our community! We are happy to assist you.

To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers your question?

This will also help other community members who may have similar questions in the future. Thank you for your participation and let us know if you need any further assistance!

Re: MergeSchema Not Working

DJey — Fri, 16 Jun 2023 09:29:52 GMT

This is resolved. Thanks, everyone!

Re: MergeSchema Not Working

DJey — Fri, 16 Jun 2023 09:34:56 GMT

@Vidula Khanna Enabling the below property resolved my issue:

spark.conf.set("spark.databricks.delta.schema.autoMerge.enabled",True)

Thanks v much!

Re: MergeSchema Not Working

g96g — Wed, 25 Oct 2023 10:34:59 GMT

sorry, can you please explain what was the problem? cause you have already that property enabled

Re: MergeSchema Not Working

Amin112 — Fri, 27 Sep 2024 03:51:35 GMT

In Databricks Runtime 15.2 and above, you can specify schema evolution in a merge statement using SQL or Delta table APIs:

MERGE WITH SCHEMA EVOLUTION INTO target
USING source
ON source.key = target.key
WHEN MATCHED THEN
UPDATE SET *
WHEN NOT MATCHED THEN
INSERT *
WHEN NOT MATCHED BY SOURCE THEN
DELETE