cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Abel_Martinez
by Contributor
  • 7299 Views
  • 9 replies
  • 6 kudos

Resolved! Why I'm getting connection timeout when connecting to MongoDB using MongoDB Connector for Spark 10.x from Databricks

I'm able to connect to MongoDB using org.mongodb.spark:mongo-spark-connector_2.12:3.0.2 and this code:df = spark.read.format("com.mongodb.spark.sql.DefaultSource").option("uri", jdbcUrl)It works well, but if I install last MongoDB Spark Connector ve...

  • 7299 Views
  • 9 replies
  • 6 kudos
Latest Reply
Kaniz
Community Manager
  • 6 kudos

Hi @Abel_Martinez, I want to express my gratitude for your effort in selecting the most suitable solution. It's great to hear that your query has been successfully resolved. Thank you for your contribution. 

  • 6 kudos
8 More Replies
sharonbjehome
by New Contributor
  • 814 Views
  • 1 replies
  • 1 kudos

Structered Streamin from MongoDB Atlas not parsing JSON correctly

HI all,I have a table in MongoDB Atlas that I am trying to read continuously to memory and then will write that file out eventually. However, when I look at the in-memory table it doesn't have the correct schema.Code here:from pyspark.sql.types impo...

image.png
  • 814 Views
  • 1 replies
  • 1 kudos
Latest Reply
Debayan
Esteemed Contributor III
  • 1 kudos

Hi @sharonbjehome​ , This has to be checked thoroughly via a support ticket, did you follow: https://docs.databricks.com/external-data/mongodb.html Also, could you please check with mongodb support, Was this working before?

  • 1 kudos
amichel
by New Contributor III
  • 4829 Views
  • 5 replies
  • 4 kudos

Resolved! Recommended way to integrate MongoDB as a streaming source

Current state:Data is stored in MongoDB Atlas which is used extensively by all servicesData lake is hosted in same AWS region and connected to MongoDB over private link Requirements:Streaming pipelines that continuously ingest, transform/analyze and ...

  • 4829 Views
  • 5 replies
  • 4 kudos
Latest Reply
Kaniz
Community Manager
  • 4 kudos

Hi @Alex Michel​ , We haven’t heard from you on the last response from the community members, and I was checking back to see if you have a resolution yet. If you have any solution, please share it with the community as it can be helpful to others. Ot...

  • 4 kudos
4 More Replies
Mr__E
by Contributor II
  • 1684 Views
  • 3 replies
  • 3 kudos

Resolved! Importing MongoDB with field names containing spaces

I am currently using a Python notebook with a defined schema to import fairly unstructured documents in MongoDB. Some of these documents have spaces in their field names. I define the schema for the MongoDB PySpark connector like the following:Struct...

  • 1684 Views
  • 3 replies
  • 3 kudos
Latest Reply
Mr__E
Contributor II
  • 3 kudos

Solution: It turns out the issue is not the schema reading in, but the fact that I am writing to Delta tables, which do not currently support spaces. So, I need to transform them prior to dumping. I've been following a pattern of reading in raw data,...

  • 3 kudos
2 More Replies
Kaniz
by Community Manager
  • 748 Views
  • 1 replies
  • 1 kudos
  • 748 Views
  • 1 replies
  • 1 kudos
Latest Reply
Ryan_Chynoweth
Honored Contributor III
  • 1 kudos

Hi Kainz, if you want to use Databricks to read data from one database and write to another database I would imagine that you would want to use the mongodb connector. Check out our docs here.

  • 1 kudos
Kaniz
by Community Manager
  • 779 Views
  • 1 replies
  • 0 kudos
  • 779 Views
  • 1 replies
  • 0 kudos
Latest Reply
Kaniz
Community Manager
  • 0 kudos

To export all collections:mongodump -d database_name -o directory_to_store_dumpsTo restore them:mongorestore -d database_name directory_backup_where_mongodb_tobe_restored

  • 0 kudos
Labels