cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results forย 
Search instead forย 
Did you mean:ย 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results forย 
Search instead forย 
Did you mean:ย 

Rename Column Name of Streaming Table in Lakeflow Spark Declarative Pipeline

guidotognini
New Contributor II

yesterday

Hi, I would like to know if it is possible to rename the name of a column of a streaming table defined in Lakeflow Spark Declarative Pipeline without having to run a Full Refresh. Could u give me any ideas on how I can achieve this?

0 Kudos
2 REPLIES 2

Ashwin_DSA
Databricks Employee
Databricks Employee

yesterday

Hi @guidotognini,

We recently covered the same question in a different post. You might want to take a look at this.

If this answer resolves your question, could you mark it as โ€œAccept as Solutionโ€? That helps other users quickly find the correct fix.

 

Regards,
Ashwin | Delivery Solution Architect @ Databricks
Helping you build and scale the Data Intelligence Platform.
***Opinions are my own***
0 Kudos

balajij8
Contributor

an hour ago

You can

  • Update pipeline code to rename old column & trigger a Incremental Update (old_column and new_column exists after it)
  • Old data will have NULL for new_column after Incremental Update. Update the table to fill new_column for such cases from old_column. old_column will have NULL for new Incremental. You can drop it in the next FULL REFRESH

You can avoid direct schema breaks for main streaming tables and do FULL REFRESH. It ensures physical consistency without conflicts.

0 Kudos
Announcements