Thanks for this, it helped me find the real issue. The data WAS bringing in the correct keys. I was building three streaming tables (group a, b, c), so I was using a loop as described here: https://spark.apache.org/docs/latest/declarative-pipelines-programming-guide.html#creating-tables-in.... However, I forgot a critical piece: passing the group into the query function definition. Since the for loop just creates the definitions of the tables, when it came to actually stream the data, group was pointing to "c" for all 3 refreshes, so all 3 tables were streaming in group c data. When I spot-checked a csv file against the table data, I saw the schema mismatch and assumed it was a checkpointing/evolution issue, but it was not.

View solution in original post