What is the best way to decode protobuf using pyspark?

WillBlock
Databricks Employee
Databricks Employee

‎10-11-2021 03:56 PM

I am using spark structured streaming to read a protobuf encoded message from the event hub. We use a lot of Delta tables, but there isn't a simple way to integrate this.

We are currently using K-SQL to transform into avro on the fly and then use Databricks to read the Avro file. Is there a better way of doing this?

Labels:
0 Kudos

jose_gonzalez
Databricks Employee
Databricks Employee

‎10-11-2021 04:38 PM

hi @Will Block​ ,

I think there is a related question being asked in the past. I think it was this one

I found this library, I hope it helps.

0 Kudos