topic Google PubSub for DLT - Error in Get Started Discussions

Google PubSub for DLT - Error

itamarwe — Sun, 14 Jul 2024 12:26:02 GMT

I'm trying to create a delta live table from a Google PubSub stream.

Unfortunately I'm getting the following error:

org.apache.spark.sql.streaming.StreamingQueryException: [PS_FETCH_RETRY_EXCEPTION] Task in pubsub fetch stage cannot be retried. Partition 0.3 in stage 3.0, TID 14. SQLSTATE: 22000

My code is below. What could explain this error and how can it be solved?

authOptions = {

"clientId" : "XXX",

"clientEmail" : "XXX",

"privateKey" : "-----BEGIN PRIVATE KEY-----\nXXX-----END PRIVATE KEY-----\n",

"privateKeyId" : "XXX"

 } 

stream=spark.readStream \

 .format("pubsub") \

 .option("subscriptionId","XXX") \

 .option("topicId","XXX") \

 .option("projectId","XXX") \

 .options(**authOptions) \

 .load()

Re: Google PubSub for DLT - Error

itamarwe — Wed, 17 Jul 2024 07:59:07 GMT

Hi @Retired_mod, it was indeed a permissions issue. Nevertheless, I must admit that the error message is slightly misleading.
Thanks.

Re: Google PubSub for DLT - Error

KriZ — Tue, 15 Oct 2024 19:05:59 GMT

Hi @itamarwe, what was the missing permissions ?

Thanks.

Re: Google PubSub for DLT - Error

sahilsagar302 — Sat, 09 Aug 2025 08:13:04 GMT

@itamarwe can you please share which permission resulted into the issue and how it got resolved