Databricks

surband

We are doing a first time implementation of data streaming reading from a partitioned pulsar topics to a delta table managed by UC. We are unable to scale the job beyond about ~ 40k msgs/sec. Beyond 40k msgs/sec , the job fails. I'd imagine Databric...

surband

I am encountering the following exception when attempting to stream data to a pulsar topic. This is a first time implementation - any ideas to resolve this is greatly appreciated.DBR: 14.3 LTS ML (includes Apache Spark 3.5.0, Scala 2.12)1 Driver64 GB...

surband

@shan_chandra I was able to install and rerun the tests and I am able to see 70k avg process/sec - and job is not crashing any longer. My goal is to see if I can achieve 500k msgs/sec. I will continue testing next week. Will databricks runtime update...

surband

@shan_chandra I was able to get the jar from https://mvnrepository.com/artifact/io.streamnative.connectors/pulsar-spark-connector_2.12/3.4.0.3But unable to install from as the wizard does not allow me to select the file - see attached. Is there an al...

surband

But that jar already seems to be available in the cluster class path, I checked in the Spark UI -->Environment -->Classpath Entries. Do you still suggest I go ahead and install? @shan_chandra

surband

@shan_chandra any suggestions?

surband

Attached Grafana screenshots

Databricks

User Stats

User Activity

Pulsar Streaming (Read) - Benchmarking Information

Failures Streaming data to Pulsar

Re: Pulsar Streaming (Read) - Benchmarking Information

Re: Pulsar Streaming (Read) - Benchmarking Information

Re: Pulsar Streaming (Read) - Benchmarking Information

Re: Pulsar Streaming (Read) - Benchmarking Information

Re: Pulsar Streaming (Read) - Benchmarking Information