- 2241 Views
- 2 replies
- 0 kudos
We have a source table that receives daily append operations, but the rows created within the last 30 days in this table can be updated or deleted. Thus, the source table is not exactly a streaming source.Our processing workflow involves performing "...
- 2241 Views
- 2 replies
- 0 kudos
Latest Reply
Hi @Pongsakorn Chairatanakul Hope everything is going great.Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please...
1 More Replies
by
gilo12
• New Contributor III
- 10272 Views
- 3 replies
- 2 kudos
I am using the following query to make an upsert:MERGE INTO my_target_table AS target
USING (SELECT MAX(__my_timestamp) AS checkpoint FROM my_source_table) AS source
ON target.name = 'some_name'
AND target.address = 'some_address'
WHEN MATCHED AN...
- 10272 Views
- 3 replies
- 2 kudos
Latest Reply
I was using a view for my_source_table, once I changed that to be a table the issue stoped.That unblocked me, but I think Databricks has a bug with using MERGE INTO from a VIEW
2 More Replies
- 8385 Views
- 1 replies
- 1 kudos
I am reading the source table which gets updated every day. It is usually append/merge with updates and is occasionally overwritten for other reasons. df = spark.readStream.schema(schema).format("delta").option("ignoreChanges", True).option('starting...
- 8385 Views
- 1 replies
- 1 kudos
Latest Reply
Hi, Could you please confirm DLT and DBR versions? Also please tag @Debayan with your next response which will notify me, Thank you!
- 9087 Views
- 9 replies
- 13 kudos
Hello guys,I'm trying to use upsert via delta lake following the documentation, but the command doesn't update or insert newlines.scenario: my source table is separated in bronze layer and updates or inserts are in silver layer.from delta.tables impo...
- 9087 Views
- 9 replies
- 13 kudos
Latest Reply
I managed to find the solution. In insert and update I was setting the target.tanks @Werner Stinckens !delta_df = DeltaTable.forPath(spark, 'dbfs:/mnt/silver/vendas/')
delta_df.alias('target').m...
8 More Replies
by
577391
• New Contributor II
- 2699 Views
- 2 replies
- 0 kudos
In my scenario, the new data coming in are the current, valid records. Any records that are not in the new data should be labeled as 'Gone", any matching records should be labeled with "Updated". And finally, any new records should be added.So in sum...
- 2699 Views
- 2 replies
- 0 kudos
Latest Reply
Detection deletions does not work out of the box.The merge statement will evaluate the incoming data against the existing data. It will not check the existing data against the incoming data.To mark deletions, you will have to specifically update tho...
1 More Replies
by
_Orc
• New Contributor
- 3955 Views
- 2 replies
- 1 kudos
Use caseRead data from source table using structured spark streaming(Round the clock).Apply transformation logic etc etc and finally merge the dataframe in the target table.If there is any failure during transformation or merge ,databricks job should...
- 3955 Views
- 2 replies
- 1 kudos
Latest Reply
Hi @Om Singh Hope you are doing well. Just wanted to check in and see if you were able to find a solution to your question?Cheers
1 More Replies
