TomRenish
New Contributor III

good morning @Kearon McNicol​ . Yes, this is working reliably. Now let me tell you why I don't like the solution. First though, some caveats...

  1. The data I'm working with is customer-service oriented. The implication here is that a given record's field values are prone to change. Simpler data such as logs have no such concern
  2. I don't consider myself to be an expert. It's plausible that my solution is dumb

Having said that, I looked more closely at what the dlt pipeline is doing and what kinds of objects are being created. When the join between the table full of IDs and the latest data takes place, the code quite explicitly calls for a table with "@dlt.table()". In the graphical view however, the outputted object is in fact a materialized view. When the materialized view is being written, it is *not* operating in an append mode! It is instead doing a complete rewrite. My data is mercifully small today with the largest table coming in at 2.3m records, but that's going to change. The full rewrite has me deeply concerned that scalability will become a problem. One solution that occurs to me is to use a spark pipeline instead and rely on keys to do MERGE operations instead.

I am of course open to suggestions on this topic.