Batch Read (spark.read.format("mongodb")) works fine.
Streaming Read (spark.readStream.format("mongodb")) runs but receives no records.
Batch Read (Works):
df = spark.read.format("mongodb")\
.option("database", database)\
.option("spark.mongodb.read.connection.uri", connectionString)\
.option("collection", collection)\
.schema(schema)\
.load()
Streaming Read (Not Receiving Records):
dfs = spark.readStream.format("mongodb")\
.option("database", database)\
.option("spark.mongodb.read.connection.uri", connectionString)\
.option("collection", collection)\
.schema(schema)\
.load()
Questions:
Does MongoDB require special settings to enable streaming?
Any known issues with MongoDB change streams on Databricks?