I am unable to display the below stream after reading it.
df= spark.readStream.format("cloudFiles")\
.option("cloudFiles.format", "csv")\
.option("header", "true")\
.option("delimiter", "\t")\
.option("inferSchema", "true")\
.option("cloudFiles.connectionString", connection_string)\
.option("cloudFiles.resourceGroup", resource_group)\
.option("cloudFiles.subscriptionId", subscription_id)\
.option("cloudFiles.tenantId", tenant_id)\
.option("cloudFiles.clientId", client_id)\
.option("cloudFiles.clientSecret", client_secret)\
.option("cloudFiles.schemaLocation", schema_folder)\
.option("cloudFiles.schemaEvolutionMode", "addNewColumns")
.load(input_folder_path)
df.display()
Below is the exception that I am getting.
com.databricks.sql.cloudfiles.errors.CloudFilesSchemaEvolutionException: Stateful streaming queries do not support schema evolution. Please set the option "cloudFiles.schemaEvolutionMode" to "rescue" or "none".<p>
I am not doing any aggregation while reading the stream so stateful stream should have not been created. Is there anything I am missing here?
Secondly, is there a command that I can fire to know whether a stream is stateful or stateless?
I am running the above command in databricks runtime 14.2.</p></em></div></div></div>
