https://community.databricks.com/t5/data-engineering/error-during-deserializing-protobuf-data/m-p/67189#M33302 <P>I am receiving protobuf data in a json attribute and along with it I receive a descriptor file.</P><P>I am using from_protobuf to deserialize the data as below,</P><P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="Sambit_S_0-1713966940987.png" style="width: 400px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/7214i7F9B674F3DBF8436/image-size/medium/is-moderation-mode/true?v=v2&amp;px=400" role="button" title="Sambit_S_0-1713966940987.png" alt="Sambit_S_0-1713966940987.png" /></span></P><P>It works most of the time but giving error when there are some recursive fields within the protobuf.</P><P>I tried the property as suggested by the spark doc&nbsp;<STRONG><A href="https://spark.apache.org/docs/latest/sql-data-sources-protobuf.html#:~:text=A%20recursive.,issues%20and%20even%20stack%20overflows" target="_blank">https://spark.apache.org/docs/latest/sql-data-sources-protobuf.html#:~:text=A%20recursive.,issues%20and%20even%20stack%20overflows</A>.</STRONG></P><P>Even if I set the property<STRONG><SPAN>recursive.fields.max.depth to its max value 10 </SPAN></STRONG>and my protobuf data contains 4 levels of recursive data, it errors out.</P><P>Any help here will be appreciated.</P><P>&nbsp;</P> Wed, 24 Apr 2024 14:01:19 GMT Sambit_S 2024-04-24T14:01:19Z https://community.databricks.com/t5/data-engineering/error-during-deserializing-protobuf-data/m-p/67189#M33302 <P>I am receiving protobuf data in a json attribute and along with it I receive a descriptor file.</P><P>I am using from_protobuf to deserialize the data as below,</P><P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="Sambit_S_0-1713966940987.png" style="width: 400px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/7214i7F9B674F3DBF8436/image-size/medium/is-moderation-mode/true?v=v2&amp;px=400" role="button" title="Sambit_S_0-1713966940987.png" alt="Sambit_S_0-1713966940987.png" /></span></P><P>It works most of the time but giving error when there are some recursive fields within the protobuf.</P><P>I tried the property as suggested by the spark doc&nbsp;<STRONG><A href="https://spark.apache.org/docs/latest/sql-data-sources-protobuf.html#:~:text=A%20recursive.,issues%20and%20even%20stack%20overflows" target="_blank">https://spark.apache.org/docs/latest/sql-data-sources-protobuf.html#:~:text=A%20recursive.,issues%20and%20even%20stack%20overflows</A>.</STRONG></P><P>Even if I set the property<STRONG><SPAN>recursive.fields.max.depth to its max value 10 </SPAN></STRONG>and my protobuf data contains 4 levels of recursive data, it errors out.</P><P>Any help here will be appreciated.</P><P>&nbsp;</P> Wed, 24 Apr 2024 14:01:19 GMT https://community.databricks.com/t5/data-engineering/error-during-deserializing-protobuf-data/m-p/67189#M33302 Sambit_S 2024-04-24T14:01:19Z