What is the best way to decode protobuf using pyspark?

WillBlock — Mon, 11 Oct 2021 22:56:47 GMT

I am using spark structured streaming to read a protobuf encoded message from the event hub. We use a lot of Delta tables, but there isn't a simple way to integrate this.

We are currently using K-SQL to transform into avro on the fly and then use Databricks to read the Avro file. Is there a better way of doing this?

Re: What is the best way to decode protobuf using pyspark?

jose_gonzalez — Mon, 11 Oct 2021 23:38:23 GMT

hi @Will Block ,

I think there is a related question being asked in the past. I think it was this one

I found this library, I hope it helps.

topic Re: What is the best way to decode protobuf using pyspark? in Data Engineering

What is the best way to decode protobuf using pyspark?

Re: What is the best way to decode protobuf using pyspark?