SchemaEvolutionMode exception in Databricks 14.2

Dikshant — Wed, 20 Mar 2024 17:08:31 GMT

I am unable to display the below stream after reading it.

df= spark.readStream.format("cloudFiles")\
.option("cloudFiles.format", "csv")\
.option("header", "true")\
.option("delimiter", "\t")\
.option("inferSchema", "true")\
.option("cloudFiles.connectionString", connection_string)\
.option("cloudFiles.resourceGroup", resource_group)\
.option("cloudFiles.subscriptionId", subscription_id)\
.option("cloudFiles.tenantId", tenant_id)\
.option("cloudFiles.clientId", client_id)\
.option("cloudFiles.clientSecret", client_secret)\
.option("cloudFiles.schemaLocation", schema_folder)\
.option("cloudFiles.schemaEvolutionMode", "addNewColumns")
.load(input_folder_path)

df.display()

Below is the exception that I am getting.

com.databricks.sql.cloudfiles.errors.CloudFilesSchemaEvolutionException: Stateful streaming queries do not support schema evolution. Please set the option "cloudFiles.schemaEvolutionMode" to "rescue" or "none".<p>

I am not doing any aggregation while reading the stream so stateful stream should have not been created. Is there anything I am missing here?

Secondly, is there a command that I can fire to know whether a stream is stateful or stateless?

I am running the above command in databricks runtime 14.2.</p></em></div></div></div>

topic SchemaEvolutionMode exception in Databricks 14.2 in Data Engineering

SchemaEvolutionMode exception in Databricks 14.2