topic Re: Schema issue when dropping a delta live table in Data Engineering

Schema issue when dropping a delta live table

ImranA — Mon, 09 Dec 2024 11:02:34 GMT

For example there is a table called "cars", if I remove the table from DLT pipeline and drop the table from catalogue. Now if I change the schema of the table, and create the table again using the same table name "cars" through the same pipeline, Why I am getting schema mismatched error?

Why it didn't remove the schema, is there any other files that we need to be aware of in the backend which doesn't get deleted automatically?

Re: Schema issue when dropping a delta live table

gchandra — Mon, 09 Dec 2024 11:05:55 GMT

Is schema evolution made true?

Or please try full Refresh.

Re: Schema issue when dropping a delta live table

ImranA — Mon, 09 Dec 2024 11:15:12 GMT

It does tell me to try full refresh in the error. I haven't changed schema evolution, it is taking the default settings.

By full refresh do you mean to do Spark.Read instead of Spark.ReadStream? (Just to add I have tried doing Spark.Read but that creates a materialised view)

Re: Schema issue when dropping a delta live table

gchandra — Mon, 09 Dec 2024 11:20:41 GMT

When executing the pipeline, click Full Refresh, which will clear the old checkpoints/schemas. In the future, please share the code snippets, too.

Re: Schema issue when dropping a delta live table

ImranA — Mon, 09 Dec 2024 17:48:00 GMT

Thanks @gchandra , "full refresh all" worked.

Does "Is schema evolution made true?" need to be added through the code or somewhere else in DLT?

Re: Schema issue when dropping a delta live table

gchandra — Mon, 09 Dec 2024 18:02:48 GMT

https://docs.databricks.com/en/ingestion/cloud-object-storage/auto-loader/schema.html#how-does-auto-loader-schema-evolution-work

https://docs.databricks.com/en/ingestion/cloud-object-storage/auto-loader/patterns.html#enable-easy-etl