Rename Column Name of Streaming Table in Lakeflow Spark Declarative Pipeline

[guidotognini]

Hi, I would like to know if it is possible to rename the name of a column of a streaming table defined in Lakeflow Spark Declarative Pipeline without having to run a Full Refresh. Could u give me any ideas on how I can achieve this?

[Ashwin_DSA]

Hi @guidotognini,

We recently covered the same question in a different post. You might want to take a look at this.

If this answer resolves your question, could you mark it as “Accept as Solution”? That helps other users quickly find the correct fix.

[balajij8]

You can

• Update pipeline code to rename old column & trigger a Incremental Update (old_column and new_column exists after it)
• Old data will have NULL for new_column after Incremental Update. Update the table to fill new_column for such cases from old_column. old_column will have NULL for new Incremental. You can drop it in the next FULL REFRESH

You can avoid direct schema breaks for main streaming tables and do FULL REFRESH. It ensures physical consistency without conflicts.