04-10-2024 11:57 AM
I am encountering the following exception when attempting to stream data to a pulsar topic. This is a first time implementation - any ideas to resolve this is greatly appreciated.
DBR: 14.3 LTS ML (includes Apache Spark 3.5.0, Scala 2.12)
Exception:
Caused by: java.lang.NoSuchMethodError: org.apache.spark.sql.types.StructType.toAttributes()Lscala/collection/Seq;
at org.apache.spark.sql.pulsar.PulsarSink.addBatch(PulsarSinks.scala:47)
at org.apache.spark.sql.execution.streaming.MicroBatchExecution.addBatch(MicroBatchExecution.scala:1236)
at org.apache.spark.sql.execution.streaming.MicroBatchExecution.$anonfun$runBatch$19(MicroBatchExecution.scala:1465)
Code:
04-11-2024 09:27 AM
@surband - The feature is in public preview within DBR. Only Read from pulsar sources are supported. we shall follow up with the engg for write support to Pulsar.
https://docs.databricks.com/en/connect/streaming/pulsar.html#stream-from-apache-pulsar
04-10-2024 12:12 PM
Hi @surband - can you please share the full error stack trace. Also, please use the compatible DBR(Spark) version instead of ML runtime. Please refer to the below document and validate if you have the necessary connector libraries added to the cluster.
https://docs.streamnative.io/hub/data-processing-pulsar-spark-3.2
04-10-2024 12:36 PM
Please see attached log files and screenshot of DBR. The ones I selected for Runtime is one of the options in the dropdown. I can't tell from DBR which version of stream native is used underneath.
04-10-2024 01:00 PM
@surband - Databricks Runtime version has a dropdown when you edit the cluster. There will be two options - Standard, ML. could you please let us know if you have spark pulsar connector added to the cluster libraries?
Per the documentation here, Structured Streaming provides exactly-once processing semantics for data read from Pulsar sources.
04-10-2024 01:26 PM
Hello Shan_chandra, Tried with Standard DBR as suggested but same result. Spark Pulsar Connector as I understand is comes preinstalled in DBR. I have not explicitly installed anything - I did not see an documentation to do the same. The attached image "streamnative-pulsar.png" is a screenshot of Environments tab - that shows it's available in class path.
04-11-2024 08:48 AM
@shan_chandra any suggestions ?
04-11-2024 09:24 AM
04-11-2024 09:27 AM
@surband - The feature is in public preview within DBR. Only Read from pulsar sources are supported. we shall follow up with the engg for write support to Pulsar.
https://docs.databricks.com/en/connect/streaming/pulsar.html#stream-from-apache-pulsar
Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.
If there isn’t a group near you, start one and help create a community that brings people together.
Request a New Group