Databricks Community

amichel · 02-22-2022

Current state:Data is stored in MongoDB Atlas which is used extensively by all servicesData lake is hosted in same AWS region and connected to MongoDB over private link Requirements:Streaming pipelines that continuously ingest, transform/analyze and ...

amichel · 11-18-2021

I want to be able to refresh tokens generated on behalf of a service principal via Token Management API, just like with any other service where OAuth is used and refresh token endpoint is available. Allowing indefinite or very long expiration for acc...

amichel · 09-28-2021

We're delivering pipelines that are mostly based on Databricks Spark Streaming, Delta Lake and Azure Event Hubs, and there's a requirement to integrate with AMQ/JMS endpoints (Request and Response queues in ActiveMQ).Is there a proven way to integrat...

amichel · 07-07-2022

Hi,Eventually we agreed on a solution to use MongoDB Atlas $out feature to export to S3 and ingest files using Databricks Autoloader as a stream.https://www.mongodb.com/developer/products/atlas/automated-continuous-data-copying-from-mongodb-to-s3/Wil...

amichel · 11-22-2021

Thanks @Hubert Dudek Appreciate your fast response.So the idea is to simulate refresh by using the token to authenticate to the api, create new token and then delete itself.Another issue with Azure specifically, is that Create Service Principal Api...

amichel · 10-06-2021

Hi @Kaniz Fatma Any insights from your team? We're facing the delivery deadline and need to decide on the optimal solution.Thank you,Alex

Databricks Community

User Stats

User Activity

Recommended way to integrate MongoDB as a streaming source

Is there a way to refresh tokens issued on behalf of service principal?

Is there a stable, ideally official JMS/ActiveMQ connector for Spark?

Re: Recommended way to integrate MongoDB as a streaming source

Re: Is there a way to refresh tokens issued on behalf of service principal?

Re: Is there a stable, ideally official JMS/ActiveMQ connector for Spark?