Schema issue when dropping a delta live table

[ImranA]

For example there is a table called "cars", if I remove the table from DLT pipeline and drop the table from catalogue. Now if I change the schema of the table, and create the table again using the same table name "cars" through the same pipeline, Why I am getting schema mismatched error?

Why it didn't remove the schema, is there any other files that we need to be aware of in the backend which doesn't get deleted automatically?

[gchandra · Accepted solution]

Is schema evolution made true?

Or please try full Refresh.

[ImranA]

It does tell me to try full refresh in the error. I haven't changed schema evolution, it is taking the default settings.

By full refresh do you mean to do Spark.Read instead of Spark.ReadStream? (Just to add I have tried doing Spark.Read but that creates a materialised view)

[gchandra · Accepted solution]

When executing the pipeline, click Full Refresh, which will clear the old checkpoints/schemas. In the future, please share the code snippets, too. 

[ImranA]

Thanks @gchandra , "full refresh all" worked. 

Does "Is schema evolution made true?" need to be added through the code or somewhere else in DLT?

[gchandra · Accepted solution]

https://docs.databricks.com/en/ingestion/cloud-object-storage/auto-loader/schema.html#how-does-auto-...

https://docs.databricks.com/en/ingestion/cloud-object-storage/auto-loader/patterns.html#enable-easy-...