MERGE in the delta table

Viren123 — Mon, 19 Dec 2022 16:29:14 GMT

Hello,

I would like to UPDATE the data in the delta table. For the same MERGE option came to my attention.

However my below code ends up with errors as shown at the end. Kindly advice what am I missing here. Thank you.

************************************************************************

from delta.tables import *

import pandas as pd

dfsource = pd.DataFrame({'BRI': {0: 'abc'},

'run_datetime': {0: '2022-12-05 16:33:22.324'}

})

tabletoupdate = DeltaTable.forName(spark, 'eda_rw.eda_log_bridge_run')

tabletoupdate.alias('target') \

.merge(

dfsource,

'target.BRI = dfsource.BRI'

'target.run_datetime = dfsource.run_datetime'

) \

.whenMatchedUpdate(set =

{

"BRI": "dfsource.BRI",

"run_datetime": "dfsource.run_datetime"

}

) \

.whenNotMatchedInsert(values =

{

"BRI": "dfsource.BRI",

"run_datetime": "dfsource.run_datetime"

}

) \

.execute()

**************************************************************************

Viren

Re: MERGE in the delta table

daniel_sahal — Thu, 22 Dec 2022 13:25:19 GMT

I think that you're mixing DataFrames spark vs. pandas.

Try creating dfSource in Spark instead of Pandas.