Databricks

sourander · 09-15-2021

Hi,Let's assume I have these things:Binary column containing protobuf-serialized dataThe .proto file including message definitionWhat different approaches have Databricks users chosen to deserialize the data? Python is the programming language that...

sourander · 04-03-2022

Hi,The ProtoFetcher is just a wrapper for the pb2 file(s) created using the protoc. As of now, I am using Scala to do the same trick (it was about 5x faster). To avoid the pickling problems, you need to build the ProtoFetcher library into a wheel fil...

sourander · 10-28-2021

In case this helps someone else, I will leave the code here for reference that I use for running the method in Databricks inside a UDF.from typing import Iterator @pandas_udf("string") def my_test_function(blobs: Iterator[pd.Series]) -> Iterator[pd...

sourander · 10-28-2021

@Dan Zafar do you have any ideas on how would I optimize the query that requires an access to a class method? Details below.I created a class ProtoFetcher that hosts various proto_pb2.py files create using protoc.tools (similarly as in the original ...

sourander · 10-11-2021

Hi @Jose Gonzalez and thank you for the reply! That library seems out of date and lacks documentation. I wonder if ScalaPB would be a better option. I don't have any scala experience, but since scala UDF's perform well with Spark. I suppose learnin...

sourander · 09-29-2021

@Kaniz Fatma, the community hasn't provided answers yet. Similar question has appeared in the Databricks Forums earlier and there are no ideas either. Do you have any ideas how to perform this deserialization efficiently in Databricks? Can we someh...

Databricks

User Stats

User Activity

Protobuf deserialization in Databricks

Re: Protobuf deserialization in Databricks

Re: Protobuf deserialization in Databricks

Re: Protobuf deserialization in Databricks

Re: Protobuf deserialization in Databricks

Re: Protobuf deserialization in Databricks