Schema evolution in Autoloader not evolving beyond version 0
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-25-2024 02:32 AM
I am working through the current version of the standard AutoLoader demo, i.e.
dbdemos.install('auto-loader')
I.e. data gets read into a dataframe, but never written to a target table.
Notebook is "01-Auto-loader-schema-evolution-Ingestion"
Notebook is "01-Auto-loader-schema-evolution-Ingestion"
Compute is a "15.4 LTS ML (includes Apache Spark 3.5.0, Scala 2.12)"
The demo works fine until schema evolution occurs the first time, i.e. there is a file
.../dbdemos_autoloader/raw_data/inferred_schema/_schemas/0
containing the evolved schema.
When I add another column that should be evolved, e.g. "new_column2" and rerun "
display(get_stream())", the new column does noit get reflected.
But if I delete .../_schemas/0, file 0 gets recreated and the new column reflects.
Alternatively, if I point
Alternatively, if I point
.option("cloudFiles.schemaLocation", f"{volume_folder}/inferred_schema")
to a new destination, e.g.
.option("cloudFiles.schemaLocation", f"{volume_folder}/inferred_schema2")
then a new _schemas/0 gets created with the expected added column present.
As summary, I never see a version "1" file getting created.
On the terminal, I can manually create a "1" file, e.g. "touch 1" - just as a test for Linux permissions.
Any thoughts on why evolution never g
As summary, I never see a version "1" file getting created.
On the terminal, I can manually create a "1" file, e.g. "touch 1" - just as a test for Linux permissions.
Any thoughts on why evolution never g
