topic Re: Reading a protobuf file in a Databricks notebook in Data Engineering

Reading a protobuf file in a Databricks notebook

Fiona — Mon, 24 Jul 2023 16:01:11 GMT

I have proto files (offline data storage) that I'd like to read from a Databricks notebook. I found this documentation (https://docs.databricks.com/structured-streaming/protocol-buffers.html), but it only covers how to read the protobuf data once the binary is already in a DataFrame. How do I read the binary data in in the first place?

Re: Reading a protobuf file in a Databricks notebook

Priyanka_Biswas — Wed, 26 Jul 2023 22:27:02 GMT

Hi @Fiona
To use Protobuf with a descriptor file, you can reference the file that is available to your compute cluster. Here are the steps to do so:

1. Import the necessary functions:

from pyspark.sql.protobuf.functions import to_protobuf, from_protobuf

2. Specify the path to the descriptor file:

descriptor_file = "/path/to/proto_descriptor.desc"

3. Use from_protobuf() to cast a binary column to a struct:

proto_events_df = input_df.select(from_protobuf(input_df.value, "BasicMessage", descFilePath=descriptor_file).alias("proto"))

4. Use to_protobuf() to cast a struct column to binary:

proto_binary_df = proto_events_df.select(to_protobuf(proto_events_df.proto, "BasicMessage", descriptor_file).alias("bytes"))

Sources:
- https://docs.databricks.com/structured-streaming/protocol-buffers.html

Re: Reading a protobuf file in a Databricks notebook

Fiona — Fri, 28 Jul 2023 13:36:50 GMT

Hi! Yeah, I think I understand everything about that, but I don't know how to create "input_df" given a file of multiple protobuf records, if that makes sense

Re: Reading a protobuf file in a Databricks notebook

StephanK — Mon, 11 Sep 2023 20:00:36 GMT

If you have proto files in offline data storage, you should be able to read them with:

input_df = spark.read.format("binaryFile").load(data_path)