Avro file format generation

[Jagan_etl]

Hi All,

We are using cluster with 9.1 run time version, I'm getting "incompatible schema exception" error while writing the data into avro file. Fields in Avro schema are more compared to dataframe output Fields. 

I tried the same in community edition with cluster 12.2 run time, in community edition same code is working . If I change the run time to 9.1 getting Incompatible schema error.

[shan_chandra]

@Jagan_etl  - Kindly provide more context if the issue is happening only in Databricks community edition. 

[Jagan_etl]

@shan_chandra Issue is happening in both community edition (cluster with 9.1 run time) and in our org databricks workspace.

In our organization we are using cluster with 9.1 run time.

Please let me know if need any other details.

[Jagan_etl]

Hi All,

Any suggestions on this.