Currently when DLT runs it observes NULL values in a column and infers that that column should be a string by default. The next time that table runs numeric values are added and it infers that it is now a numeric column. DLT tries to merge these two columns and errors because the column schemas cannot be merged.
I understand why the column was set to a string since it had to be set to something and a NULL value could indicate any type of field. This makes sense. The desired behavior should be that when the DLT pipeline runs It sees the schema mismatch and checks the columns. If the persisted column shows all null values and the new column being brought in has values, then the persisted column should be dropped automatically and recreated with the schema from the new column.
Currently without this behavior pipelines will potentially break every time new data is added if the column originally was null prior.
