topic Avro file format generation in Data Engineering

Avro file format generation

Jagan_etl — Tue, 27 Feb 2024 15:32:24 GMT

Hi All,

We are using cluster with 9.1 run time version, I'm getting "incompatible schema exception" error while writing the data into avro file. Fields in Avro schema are more compared to dataframe output Fields.

I tried the same in community edition with cluster 12.2 run time, in community edition same code is working . If I change the run time to 9.1 getting Incompatible schema error.

Re: Avro file format generation

shan_chandra — Tue, 27 Feb 2024 16:48:49 GMT

@Jagan_etl - Kindly provide more context if the issue is happening only in Databricks community edition.

Re: Avro file format generation

Jagan_etl — Tue, 27 Feb 2024 18:12:17 GMT

@shan_chandra Issue is happening in both community edition (cluster with 9.1 run time) and in our org databricks workspace.

In our organization we are using cluster with 9.1 run time.

Please let me know if need any other details.

Re: Avro file format generation

Jagan_etl — Mon, 04 Mar 2024 13:28:21 GMT

Hi All,

Any suggestions on this.