I have gone through this kind of issue sometime back! Adding the column back doesn't undo the schema change from the streaming engine's perspective.

When you dropped the column, Delta recorded that as a specific transaction in the table's log (say, version N). When you added it back, that's a separate transaction (version N+1). The streaming checkpoint remembers exactly which version it last processed. When it tries to advance through the log, it hits version N, then drops and fails right there. It doesn't look ahead to see that you re-added the column in N+1. It just sees a non-additive schema change at N and stops.

Think of it this way: the error isn't about the current schema of the source table. It's about the history of schema changes the stream has to replay through. The stream processes the Delta log sequentially, and it can't skip over an incompatible change even if a later transaction "fixes" it.

So even though your source table's current schema now matches what the checkpoint expects, the problematic drop transaction is still sitting in the log between the checkpoint's last processed version and now.

To get past this you'll need to do a full refresh of the affected streaming table in SDP. That resets the checkpoint entirely and starts reading from the current state of the source table with its current schema. In the pipeline UI, you can select just that specific table for a full refresh rather than refreshing everything.

Going forward, if you need to temporarily remove a column from the source, a safer approach is to create a view on top of the source table that excludes the column, and have your SDP pipeline read from that view instead. That way the underlying Delta table's schema stays stable from the stream's perspective.

Hope this helps! If it did, please mark it as “Accept as Solution” so others can easily find it.