topic MongoDB Streaming Not Receiving Records in Databricks in Data Engineering

MongoDB Streaming Not Receiving Records in Databricks

vidya_kothavale — Fri, 14 Feb 2025 09:52:37 GMT

Batch Read (spark.read.format("mongodb")) works fine.
Streaming Read (spark.readStream.format("mongodb")) runs but receives no records.

Batch Read (Works):
df = spark.read.format("mongodb")\
.option("database", database)\
.option("spark.mongodb.read.connection.uri", connectionString)\
.option("collection", collection)\
.schema(schema)\
.load()

Streaming Read (Not Receiving Records):

dfs = spark.readStream.format("mongodb")\
.option("database", database)\
.option("spark.mongodb.read.connection.uri", connectionString)\
.option("collection", collection)\
.schema(schema)\
.load()

Questions:

Does MongoDB require special settings to enable streaming?
Any known issues with MongoDB change streams on Databricks?

Re: MongoDB Streaming Not Receiving Records in Databricks

Alberto_Umana — Fri, 14 Feb 2025 12:55:08 GMT

Hello @vidya_kothavale,

MongoDB requires the use of change streams to enable streaming. Change streams allow applications to access real-time data changes without polling the database. Ensure that your MongoDB instance is configured to support change streams. Change Streams are available for replica sets and sharded clusters in MongoDB 3.6 and later

Re: MongoDB Streaming Not Receiving Records in Databricks

vidya_kothavale — Fri, 14 Feb 2025 13:24:13 GMT

@Alberto_Umana Currently, only the records received after streaming started are available; the previous records are missing. Is there any additional steps required?