Schema issue when dropping a delta live table

ImranA
Contributor

‎12-09-2024 03:02 AM

For example there is a table called "cars", if I remove the table from DLT pipeline and drop the table from catalogue. Now if I change the schema of the table, and create the table again using the same table name "cars" through the same pipeline, Why I am getting schema mismatched error?

Why it didn't remove the schema, is there any other files that we need to be aware of in the backend which doesn't get deleted automatically?

0 Kudos

gchandra
Databricks Employee
Databricks Employee

‎12-09-2024 03:05 AM

Is schema evolution made true?

Or please try full Refresh.



~

View solution in original post

ImranA
Contributor
In response to gchandra

‎12-09-2024 03:15 AM

It does tell me to try full refresh in the error. I haven't changed schema evolution, it is taking the default settings.

By full refresh do you mean to do Spark.Read instead of Spark.ReadStream? (Just to add I have tried doing Spark.Read but that creates a materialised view)

0 Kudos

gchandra
Databricks Employee
Databricks Employee

‎12-09-2024 03:20 AM

When executing the pipeline, click Full Refresh, which will clear the old checkpoints/schemas. In the future, please share the code snippets, too. 

gchandra_0-1733743183680.png

 



~

View solution in original post

ImranA
Contributor

‎12-09-2024 09:48 AM

Thanks @gchandra , "full refresh all" worked. 

Does "Is schema evolution made true?" need to be added through the code or somewhere else in DLT?

0 Kudos