cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
Abel_Martinez
by • Contributor
  • 11028 Views
  • 9 replies
  • 6 kudos

Resolved! Why I'm getting connection timeout when connecting to MongoDB using MongoDB Connector for Spark 10.x from Databricks

I'm able to connect to MongoDB using org.mongodb.spark:mongo-spark-connector_2.12:3.0.2 and this code:df = spark.read.format("com.mongodb.spark.sql.DefaultSource").option("uri", jdbcUrl)It works well, but if I install last MongoDB Spark Connector ve...

Data Engineering
Reply
  • 11028 Views
  • 9 replies
  • 6 kudos
View Replies
Latest Reply
Kaniz_Fatma
Community Manager
  • 6 kudos

Hi @Abel_Martinez, I want to express my gratitude for your effort in selecting the most suitable solution. It's great to hear that your query has been successfully resolved. Thank you for your contribution. 

  • 6 kudos
8 More Replies
sharonbjehome
by • New Contributor
  • 1249 Views
  • 1 replies
  • 1 kudos

Structered Streamin from MongoDB Atlas not parsing JSON correctly

HI all,I have a table in MongoDB Atlas that I am trying to read continuously to memory and then will write that file out eventually. However, when I look at the in-memory table it doesn't have the correct schema.Code here:from pyspark.sql.types impo...

image.png
Data Engineering
Reply
  • 1249 Views
  • 1 replies
  • 1 kudos
View Replies
Latest Reply
Debayan
Esteemed Contributor III
  • 1 kudos

Hi @sharonbjehome​ , This has to be checked thoroughly via a support ticket, did you follow: https://docs.databricks.com/external-data/mongodb.html Also, could you please check with mongodb support, Was this working before?

  • 1 kudos
amichel
by • New Contributor III
  • 6389 Views
  • 5 replies
  • 4 kudos

Resolved! Recommended way to integrate MongoDB as a streaming source

Current state:Data is stored in MongoDB Atlas which is used extensively by all servicesData lake is hosted in same AWS region and connected to MongoDB over private link Requirements:Streaming pipelines that continuously ingest, transform/analyze and ...

Data Engineering
Reply
  • 6389 Views
  • 5 replies
  • 4 kudos
View Replies
Latest Reply
Kaniz_Fatma
Community Manager
  • 4 kudos

Hi @Alex Michel​ , We haven’t heard from you on the last response from the community members, and I was checking back to see if you have a resolution yet. If you have any solution, please share it with the community as it can be helpful to others. Ot...

  • 4 kudos
4 More Replies
Mr__E
by • Contributor II
  • 2396 Views
  • 3 replies
  • 3 kudos

Resolved! Importing MongoDB with field names containing spaces

I am currently using a Python notebook with a defined schema to import fairly unstructured documents in MongoDB. Some of these documents have spaces in their field names. I define the schema for the MongoDB PySpark connector like the following:Struct...

Data Engineering
Reply
  • 2396 Views
  • 3 replies
  • 3 kudos
View Replies
Latest Reply
Mr__E
Contributor II
  • 3 kudos

Solution: It turns out the issue is not the schema reading in, but the fact that I am writing to Delta tables, which do not currently support spaces. So, I need to transform them prior to dumping. I've been following a pattern of reading in raw data,...

  • 3 kudos
2 More Replies
Kaniz_Fatma
by • Community Manager
  • 1145 Views
  • 1 replies
  • 1 kudos

Resolved! How do I copy a database from one MongoDB server to another?

Data Engineering
Reply
  • 1145 Views
  • 1 replies
  • 1 kudos
View Replies
Latest Reply
Ryan_Chynoweth
Esteemed Contributor
  • 1 kudos

Hi Kainz, if you want to use Databricks to read data from one database and write to another database I would imagine that you would want to use the mongodb connector. Check out our docs here.

  • 1 kudos
Kaniz_Fatma
by • Community Manager
  • 1145 Views
  • 1 replies
  • 0 kudos

Resolved! How to export all collections in MongoDB?

Data Engineering
Reply
  • 1145 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

To export all collections:mongodump -d database_name -o directory_to_store_dumpsTo restore them:mongorestore -d database_name directory_backup_where_mongodb_tobe_restored

  • 0 kudos
Labels