.overwriteschema + writestream
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-20-2023 03:57 AM
HelloI have issue with overwriting schema while using writestream - I do not receive any error - however schema remain unchanged
Below example
df_abc = spark.readstream
.format("cloudFiles")
.option("cloudFiles.format", "parquet")
.option"cloudFiles.schemaLocation", chklocat )
.load(deltatbl)
df_abc = df_abc.withColumn("columna", col("columna").cast("timestamp"))
write = df_abc.writestream
.outputMode("append")
.option("checkpointLocation",chklocat)
.trigger(availableNow=True)
.option("overwriteSchema", "true")
.toTable(dbname + "." + tblname)