Failures Streaming data to Pulsar

surband · ‎04-10-2024

I am encountering the following exception when attempting to stream data to a pulsar topic. This is a first time implementation - any ideas to resolve this is greatly appreciated.

DBR: 14.3 LTS ML (includes Apache Spark 3.5.0, Scala 2.12)

1 Driver64 GB Memory, 16 Cores

Runtime14.3.x-cpu-ml-scala2.12

Exception:

Caused by: java.lang.NoSuchMethodError: org.apache.spark.sql.types.StructType.toAttributes()Lscala/collection/Seq;
at org.apache.spark.sql.pulsar.PulsarSink.addBatch(PulsarSinks.scala:47)
at org.apache.spark.sql.execution.streaming.MicroBatchExecution.addBatch(MicroBatchExecution.scala:1236)
at org.apache.spark.sql.execution.streaming.MicroBatchExecution.$anonfun$runBatch$19(MicroBatchExecution.scala:1465)
Code:

val query = sourceDF

.select( to_json( struct(col("*")) ) .alias("value") )

.writeStream

.format("pulsar")

.option("service.url", pulsarServiceUrl)

.option("topic", pulsarTopic)

.option("checkpointLocation", checkpointLocation)

.trigger(Trigger.ProcessingTime("10 seconds"))

.start()