Hi,
Eventually we agreed on a solution to use MongoDB Atlas $out feature to export to S3 and ingest files using Databricks Autoloader as a stream.
https://www.mongodb.com/developer/products/atlas/automated-continuous-data-copying-from-mongodb-to-s...
Will need to try the new connector too as suggested here.