topic Re: Failures Streaming data to Pulsar in Data Engineering

Failures Streaming data to Pulsar

surband — Wed, 10 Apr 2024 18:57:02 GMT

I am encountering the following exception when attempting to stream data to a pulsar topic. This is a first time implementation - any ideas to resolve this is greatly appreciated.

DBR: 14.3 LTS ML (includes Apache Spark 3.5.0, Scala 2.12)

1 Driver64 GB Memory, 16 Cores

Runtime14.3.x-cpu-ml-scala2.12

Exception:

Caused by: java.lang.NoSuchMethodError: org.apache.spark.sql.types.StructType.toAttributes()Lscala/collection/Seq;
at org.apache.spark.sql.pulsar.PulsarSink.addBatch(PulsarSinks.scala:47)
at org.apache.spark.sql.execution.streaming.MicroBatchExecution.addBatch(MicroBatchExecution.scala:1236)
at org.apache.spark.sql.execution.streaming.MicroBatchExecution.$anonfun$runBatch$19(MicroBatchExecution.scala:1465)
Code:

val query = sourceDF

.select( to_json( struct(col("*")) ) .alias("value") )

.writeStream

.format("pulsar")

.option("service.url", pulsarServiceUrl)

.option("topic", pulsarTopic)

.option("checkpointLocation", checkpointLocation)

.trigger(Trigger.ProcessingTime("10 seconds"))

.start()

Re: Failures Streaming data to Pulsar

shan_chandra — Wed, 10 Apr 2024 19:12:00 GMT

Hi @surband - can you please share the full error stack trace. Also, please use the compatible DBR(Spark) version instead of ML runtime. Please refer to the below document and validate if you have the necessary connector libraries added to the cluster.

https://docs.streamnative.io/hub/data-processing-pulsar-spark-3.2

Re: Failures Streaming data to Pulsar

surband — Wed, 10 Apr 2024 19:36:55 GMT

Please see attached log files and screenshot of DBR. The ones I selected for Runtime is one of the options in the dropdown. I can't tell from DBR which version of stream native is used underneath.

Re: Failures Streaming data to Pulsar

shan_chandra — Wed, 10 Apr 2024 20:00:58 GMT

@surband - Databricks Runtime version has a dropdown when you edit the cluster. There will be two options - Standard, ML. could you please let us know if you have spark pulsar connector added to the cluster libraries?

Per the documentation here, Structured Streaming provides exactly-once processing semantics for data read from Pulsar sources.

Re: Failures Streaming data to Pulsar

surband — Wed, 10 Apr 2024 20:26:49 GMT

Hello Shan_chandra, Tried with Standard DBR as suggested but same result. Spark Pulsar Connector as I understand is comes preinstalled in DBR. I have not explicitly installed anything - I did not see an documentation to do the same. The attached image "streamnative-pulsar.png" is a screenshot of Environments tab - that shows it's available in class path.

Re: Failures Streaming data to Pulsar

surband — Thu, 11 Apr 2024 15:48:13 GMT

@shan_chandra any suggestions ?

Re: Failures Streaming data to Pulsar

surband — Thu, 11 Apr 2024 16:24:13 GMT

Logs attached

Re: Failures Streaming data to Pulsar

shan_chandra — Thu, 11 Apr 2024 16:27:06 GMT

@surband - The feature is in public preview within DBR. Only Read from pulsar sources are supported. we shall follow up with the engg for write support to Pulsar.

https://docs.databricks.com/en/connect/streaming/pulsar.html#stream-from-apache-pulsar