topic Re: How can I write Pandas dataframe into avro in Data Engineering

How can I write Pandas dataframe into avro

sobhan — Wed, 07 Aug 2019 10:39:28 GMT

I am trying to write Pandas core dataframe into avro format as below. But I get the following error:

AttributeError: 'DataFrame' object has no attribute 'write'

I have tried several options as below:

df_2018_pd.write.format("com.databricks.spark.avro").save("s3://folder/sub-folder/avro") df_2018_pd.write.format("avro").save("s3://folder/sub-folder/avro") df_2018_pd.write().format("com.databricks.spark.avro").save("s3://folder/sub-folder/avro") df_2018_pd.format("avro").save("s3://folder/sub-folder/avro")

All of the above gives error.

Any help would be appreciated.

Re: How can I write Pandas dataframe into avro

JosieMundi — Fri, 09 Aug 2019 12:00:56 GMT

Not sure if you could use something like:

https://github.com/ynqa/pandavro

they have a to_avro which writes pandas to avro

Re: How can I write Pandas dataframe into avro

sobhan — Fri, 09 Aug 2019 14:59:13 GMT

Yes i saw that pandavro pkg and installed with pip3 but still it did not resolve. So I had to convert into JSON dictionary object and proceed from there. I solved the problem not using Pandas dataframe into AVRO. May be some useful Pandas utility for this will come in future

Thanks for your response.

Re: How can I write Pandas dataframe into avro

Brayden_Cook — Wed, 28 Aug 2019 09:15:34 GMT

Very complicated question. I think you can get your answer on online sites. There are many online providers like managements writing solutions whose experts provide online help for every type of research paper. I got a lot of assistance from them. Nowadays it has become very easy to solve any type of questions with the help online websites.