Databricks Community

thewizard · 12-11-2023

Introduction Google Pub/Sub is a fully managed messaging service with pub/sub semantics that is widely used by customers on Google Cloud Platform (GCP), often as an alternative to Kafka. Structured Streaming has long had support for other cloud plat...

thewizard · 06-20-2024

Apologies for late reply, I was at DAIS. You should not receive a message to be dropped with dropDuplicatesWithinWatermark (except where it is a genuine duplicate, based on the specific keys). So I would raise a support ticket if the problem persists...

thewizard · 06-07-2024

I don't know enough about your configuration and use case to really answer this. There shouldn't be duplicates written by the pub/sub connector, as they are deduped in RocksDB. However they are deduped using the messageId field, which may be differen...

Databricks Community

User Stats

User Activity

High Throughput ‘Exactly Once’ Streaming from Google Pub/Sub with Structured Streaming on Databricks

Re: High Throughput ‘Exactly Once’ Streaming from Google Pub/Sub with Structured Streaming on Databr

Re: High Throughput ‘Exactly Once’ Streaming from Google Pub/Sub with Structured Streaming on Databr