Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
02-01-2023 01:33 AM
It can be solved, for example, by specifying the schema with ID as string.
If you don't want to specify schema, better would be autoloader as you can specify hint:
spark.readStream
.format("cloudFiles")
.option("cloudFiles.format", "csv")
.option("cloudFiles.schemaLocation", checkpoint_path)
.option("cloudFiles.schemaHints", "ID String")
.option("inferSchema", True)
.option("mergeSchema", True)
.load(folder)
My blog: https://databrickster.medium.com/