robwma
New Contributor III

Another option if you'd like to use Spark as the ingestion is to use the new Spark Connector V10.0 which support Spark Structured Streaming. https://www.mongodb.com/developer/languages/python/streaming-data-apache-spark-mongodb/.

If you use Kafka, the MongoDB Connector when used as a source creates a change stream under the covers and has a flag, "copy.existing" which will copy the existing data first then start the stream of data.

View solution in original post