cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
William_Scardua
by • Valued Contributor
  • 1542 Views
  • 2 replies
  • 1 kudos

Cosmos DB Connector for 12.1 Cluster and above

Hi guys,​You know what`s the version suporte the databricks cluster version above 12.1 ?my cluster:error: Thank you

databricks-cluster cosmosdb-connector-erro
Data Engineering
Reply
  • 1542 Views
  • 2 replies
  • 1 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 1 kudos

Hi @William Scardua​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answe...

  • 1 kudos
1 More Replies
parulpaul
by • New Contributor III
  • 2342 Views
  • 5 replies
  • 7 kudos

Trying to stream data from cosmosDb for mongoAPI (readstream/writeStream) into databricks getting error "java.lang.UnsupportedOperationException: Data source mongodb does not support microbatch processing"

Data Engineering
Reply
  • 2342 Views
  • 5 replies
  • 7 kudos
View Replies
Latest Reply
parulpaul
New Contributor III
  • 7 kudos

No solution found

  • 7 kudos
4 More Replies
ImAbhishekTomar
by • New Contributor III
  • 2648 Views
  • 2 replies
  • 1 kudos

Resolved! Trying to Flatten My Json using CosmosDB Spark connector - Azure Databricks

Hi,Using the below cosmos DB query it is possible to achieve the expected output, but how can I do the same with spark SQL in Databricks.COSMOSDB QUERY : select c.ReportId,c.ReportName,i.price,p as provider from c join i in in_network join p in i.pr...

Data Engineering
Reply
  • 2648 Views
  • 2 replies
  • 1 kudos
View Replies
Latest Reply
Hubert-Dudek
Esteemed Contributor III
  • 1 kudos

Hi @Abhishek Tomar​ , If you want to get it from Cosmos DB, use the connector with a custom query https://github.com/Azure/azure-cosmosdb-sparkIf you want to have JSON imported directly by databricks/spark, please go with the below solution:SELECT ...

  • 1 kudos
1 More Replies
Sandesh87
by • New Contributor III
  • 1748 Views
  • 3 replies
  • 2 kudos

Resolved! log error to cosmos db

Objective:- Retrieve objects from an S3 bucket using a 'get' api call, write the retrieved object to azure datalake and in case of errors like 404s (object not found) write the error message to cosmos DB"my_dataframe" consists of the a column (s3Obje...

Data Engineering
Reply
  • 1748 Views
  • 3 replies
  • 2 kudos
View Replies
Latest Reply
User16763506477
Contributor III
  • 2 kudos

Hi @Sandesh Puligundla​  issue is that you are using spark context inside foreachpartition. You can create a dataframe only on the spark driver. Few stack overflow references https://stackoverflow.com/questions/46964250/nullpointerexception-creatin...

  • 2 kudos
2 More Replies
Labels