Hubert-Dudek
Databricks MVP

It can be solved, for example, by specifying the schema with ID as string.

If you don't want to specify schema, better would be autoloader as you can specify hint:

spark.readStream
  .format("cloudFiles")
  .option("cloudFiles.format", "csv")
  .option("cloudFiles.schemaLocation", checkpoint_path)
  .option("cloudFiles.schemaHints", "ID String")
  .option("inferSchema", True)
  .option("mergeSchema", True)
  .load(folder)


My blog: https://databrickster.medium.com/