Google PubSub for DLT - Error

[itamarwe]

I'm trying to create a delta live table from a Google PubSub stream.

Unfortunately I'm getting the following error:

org.apache.spark.sql.streaming.StreamingQueryException: [PS_FETCH_RETRY_EXCEPTION] Task in pubsub fetch stage cannot be retried. Partition 0.3 in stage 3.0, TID 14. SQLSTATE: 22000

My code is below. What could explain this error and how can it be solved?

authOptions = {

"clientId" : "XXX",

"clientEmail" : "XXX",

"privateKey" : "-----BEGIN PRIVATE KEY-----\nXXX-----END PRIVATE KEY-----\n",

"privateKeyId" : "XXX"

 } 

stream=spark.readStream \

 .format("pubsub") \

 .option("subscriptionId","XXX") \

 .option("topicId","XXX") \

 .option("projectId","XXX") \

 .options(**authOptions) \

 .load()

[Kaniz_Fatma · Accepted solution]

Hi @itamarwe, Could you please double-check your authentication credentials (authOptions). Please ensure that the client ID, client email, private key, and private key ID are correctly specified and verify that the service account associated with these credentials has the necessary permissions to access the PubSub topic and subscription.

[itamarwe]

Hi @Kaniz_Fatma, it was indeed a permissions issue. Nevertheless, I must admit that the error message is slightly misleading.
Thanks.

[Kaniz_Fatma]

Cheers, Glad to hear that.