Databricks

Kearon · 04-06-2023

OK. So I think I'm probably missing the obvious and tying myself in knots here.Here is the scenario:batch datasets arrive in json format in an Azure data lakeeach batch is a complete set of "current" records (the complete table)these are processed us...

Kearon · 03-09-2023

I have a streaming pipeline that ingests json files from a data lake using autoloader. These files are dumped there periodically. Mostly the files contain duplicate data, but there are occasional changes. I am trying to process these files into a dat...

Kearon · 04-25-2023

Hi @Suteja Kanuri ,DBR 12.1 supports and adds to MERGE functionality.My guess is that the problem lies in me trying to use the table feeding the SCD table to identify changes - so it is somewhat circular - and conflicts could arise.Trying to get my ...

Kearon · 04-17-2023

Hi @Suteja Kanuri My pipeline is running DBR 12.1.Is it because we're trying to do a merge with tables that are streaming live tables?In case I'm missing something, here is my pipeline config:{ "id": "**********************************", "cl...

Kearon · 04-14-2023

It seems my DLT pipeline doesn't support the merge operation. Do I need to change a setting or update my runtime?

Kearon · 04-12-2023

Hi @Suteja Kanuri,Thanks again.I get this error: AttributeError: 'Dataset' object has no attribute 'merge',Map(),Map(),List(),List(),Map())

Kearon · 04-12-2023

@Suteja Kanuri . Thank you once again. I'm reviewing your reply now.I realise I left data lake account info in my code I copied. While it's not a major risk, would you mind removing it from your reply please? Thank you.

Databricks

User Stats

User Activity

Process batches in a streaming pipeline - identifying deletes

Databricks Delta Live Table stored as SCD 2 is creating new records when no data changes. How do I stop this?

Re: Process batches in a streaming pipeline - identifying deletes

Re: Process batches in a streaming pipeline - identifying deletes

Re: Process batches in a streaming pipeline - identifying deletes

Re: Process batches in a streaming pipeline - identifying deletes

Re: Process batches in a streaming pipeline - identifying deletes