Hello Brahma,
Thank you for your response. To answer your suggestions:

1) cloudFiles.schemaEvolutionMode: It is default behaviour, but I have also added it explicitly
.option("cloudFiles.schemaEvolutionMode", "addNewColumns")

2) "try pointing to a new schema location to reset schema" - I have repointed it as a test, and a new file 0 gets created. I have also just renamed 0 to zero in the terminal, and a new file 0 gets created. In both cases schema evolution picked up the new column(s).

3) "file access permissions" they are always rwxrwxrwx for the files, and drwxrwxrwx fir the directories.

4) Other: IO have also tried with fresh cluster, and with different locations. I have tested with the latest runtime version via "use_current_cluster=True", and also with the cluster version that it creates itself.

Extra info:

It definitely reads the latest version of the evolution file. I have edited 0 (or 1) with vi, and changed the first line "v1" to "v2". An error gets thrown about not being happy with "v2". But also a second error with "UnknownFieldException" that is expected in the demo. This error does not get raised in my normal testing.

I managed to get evolution to work as expected, but once only, This involved renaming 0 to zero, adding new column, copying the new 0 to 1, adding new columns, and after that, just adding new columns with no fiddling inbetween. But I reset the demo and could not get it working again.

I wonder if anyone else has the demo up and running, and could confirm whether they get the same issue or not.
Basically the frames called are:
f2 to reset the demo, with $reset_all_data=true
f11 to do the initial inference
Then playing with
f16 to add a new column name each time
f17 to load and display the dataframe to check whether the new column got picked up after a "UnknownFieldException" message.

Kind regards - Roelof