topic Re: Unable to stream from google pub/sub in Data Engineering

Unable to stream from google pub/sub

210573 — Wed, 18 May 2022 22:24:56 GMT

I am trying to run below for subscribing to a pubsub but this code is throwing this exception

java.lang.NoClassDefFoundError: org/apache/spark/sql/sources/v2/DataSourceV2

I have tried using all versions of https://mvnrepository.com/artifact/com.google.cloud/pubsublite-spark-sql-streaming no luck so far.

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName('Simple Pub/Sub Lite Read').getOrCreate()

df = spark.readStream \

.format("pubsublite") \

.option("pubsublite.subscription", "My subscription path") \

.option("gcp.credentials.key", "my gcp credential").load()

df.show(10, False)

Re: Unable to stream from google pub/sub

-werners- — Thu, 19 May 2022 07:23:05 GMT

can you retry without creating a sparksession? As databricks provides one for you.

Re: Unable to stream from google pub/sub

Noopur_Nigam — Thu, 02 Jun 2022 03:37:49 GMT

Hi @cloud user As of now, we do not have structured streaming support with Pub/Sub. Below are the supported sources with structured streaming:

https://docs.gcp.databricks.com/spark/latest/structured-streaming/data-sources.html

Re: Unable to stream from google pub/sub

Ajay-Pandey — Fri, 18 Aug 2023 05:02:53 GMT

Hi @210573

Databricks now start supporting pub/sub streaming natively now you can start using pubsub streaming for your use case for more info visit below official URL -

PUB/SUB with Databricks

Re: Unable to stream from google pub/sub

davidkhala-ms — Sun, 09 Feb 2025 08:12:13 GMT

I see some issues from using pubsub as source.

in the writeStream, both .foreach or .foreachBatch cannot work to be called when stream data arrives