Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-21-2022 10:44 AM
Another option if you'd like to use Spark as the ingestion is to use the new Spark Connector V10.0 which support Spark Structured Streaming. https://www.mongodb.com/developer/languages/python/streaming-data-apache-spark-mongodb/.
If you use Kafka, the MongoDB Connector when used as a source creates a change stream under the covers and has a flag, "copy.existing" which will copy the existing data first then start the stream of data.