I am trying to run below for subscribing to a pubsub but this code is throwing this exception

java.lang.NoClassDefFoundError: org/apache/spark/sql/sources/v2/DataSourceV2

I have tried using all versions of https://mvnrepository.com/artifact/com.google.cloud/pubsublite-spark-sql-streaming no luck so far.

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName('Simple Pub/Sub Lite Read').getOrCreate()

df = spark.readStream \

 .format("pubsublite") \

 .option("pubsublite.subscription", "My subscription path") \

 .option("gcp.credentials.key", "my gcp credential").load()

df.show(10, False)