Databricks Community

tom_shaffner · 05-18-2022

In streaming flows I periodically get a "Detected a data update" error. This error generally seem to indicate that something has changed in the source table schema, but it's not immediately apparent what. In one case yesterday I pulled the source tab...

tom_shaffner · 05-18-2022

Short version: I need a way to take only the most recent record from a variable number of tables in a stream. This is a relatively easy problem in sql or python pandas (group by and take the newest) but in a stream I keep hitting blocks. I could do i...

tom_shaffner · 04-25-2022

I'm trying to use delta live tables, but if I import even the example notebooks I get a warning saying `ModuleNotFoundError: No module named 'dlt'`. If I try and install via pip it attempts to install a deep learning framework of some sort.I checked ...

tom_shaffner · 05-24-2022

@Håkon Åmdal to be clear, I think what you're suggesting is in fact the only approach that might work. i.e. an upsert for each data source, sequentially.This being the only way seems to be a spark limitation; as the answer at apache spark - Structur...

tom_shaffner · 05-24-2022

Yes, "Append" mode is doing a Upsert/Merge in pyspark, which is the same thing as the SQL Merge Into I believe; that's what gave me the above error.And I had another version that did a Merge-Into by individual data source, but I spoke to Databricks a...

tom_shaffner · 05-19-2022

@Kaniz Fatma , Thanks, that helps. I was assuming this warning indicated a schema evolution, and based on what you say it likely wasn't and I just have to turn on IgnoreChanges any time I have a stream from a table that receives updates/upserts.To b...

tom_shaffner · 04-28-2022

Got it. That helps, thanks.That could maybe be clearer in the documentation; it wasn't immediately clear to me that I couldn't run this outside a normal notebook environment. From the documentation it sounded like I could develop that way and then se...

Databricks Community

User Stats

User Activity

"Detected a data update", what changed?

How to take only the most recent record from a variable number of tables in a stream

Is there some form of enablement required to use Delta Live Tables (DLT)?

Re: How to take only the most recent record from a variable number of tables in a stream

Re: How to take only the most recent record from a variable number of tables in a stream

Re: "Detected a data update", what changed?

Re: Is there some form of enablement required to use Delta Live Tables (DLT)?