3 weeks ago
I am encountering the following exception when attempting to stream data to a pulsar topic. This is a first time implementation - any ideas to resolve this is greatly appreciated.
DBR: 14.3 LTS ML (includes Apache Spark 3.5.0, Scala 2.12)
Exception:
Caused by: java.lang.NoSuchMethodError: org.apache.spark.sql.types.StructType.toAttributes()Lscala/collection/Seq;
at org.apache.spark.sql.pulsar.PulsarSink.addBatch(PulsarSinks.scala:47)
at org.apache.spark.sql.execution.streaming.MicroBatchExecution.addBatch(MicroBatchExecution.scala:1236)
at org.apache.spark.sql.execution.streaming.MicroBatchExecution.$anonfun$runBatch$19(MicroBatchExecution.scala:1465)
Code:
3 weeks ago
@surband - The feature is in public preview within DBR. Only Read from pulsar sources are supported. we shall follow up with the engg for write support to Pulsar.
https://docs.databricks.com/en/connect/streaming/pulsar.html#stream-from-apache-pulsar
3 weeks ago
Hi @surband - can you please share the full error stack trace. Also, please use the compatible DBR(Spark) version instead of ML runtime. Please refer to the below document and validate if you have the necessary connector libraries added to the cluster.
https://docs.streamnative.io/hub/data-processing-pulsar-spark-3.2
3 weeks ago
3 weeks ago
@surband - Databricks Runtime version has a dropdown when you edit the cluster. There will be two options - Standard, ML. could you please let us know if you have spark pulsar connector added to the cluster libraries?
Per the documentation here, Structured Streaming provides exactly-once processing semantics for data read from Pulsar sources.
3 weeks ago
Hello Shan_chandra, Tried with Standard DBR as suggested but same result. Spark Pulsar Connector as I understand is comes preinstalled in DBR. I have not explicitly installed anything - I did not see an documentation to do the same. The attached image "streamnative-pulsar.png" is a screenshot of Environments tab - that shows it's available in class path.
3 weeks ago
@shan_chandra any suggestions ?
3 weeks ago
3 weeks ago
@surband - The feature is in public preview within DBR. Only Read from pulsar sources are supported. we shall follow up with the engg for write support to Pulsar.
https://docs.databricks.com/en/connect/streaming/pulsar.html#stream-from-apache-pulsar
Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections.
Click here to register and join today!
Engage in exciting technical discussions, join a group with your peers and meet our Featured Members.