topic Re: Expose delta table data to Salesforce - odata? in Get Started Discussions

Expose delta table data to Salesforce - odata?

Ruby8376 — Thu, 21 Mar 2024 18:16:19 GMT

HI Looking for suggestiongs to stream on demand data from databricks delta tables to salesforce.

Is odata a good option?

Re: Expose delta table data to Salesforce - odata?

Ruby8376 — Mon, 25 Mar 2024 14:06:58 GMT

@-werners- @Anonymous can u help?

Re: Expose delta table data to Salesforce - odata?

-werners- — Mon, 25 Mar 2024 15:09:19 GMT

So do you want Databricks to push data into Salesforce or do you want Salesforce to pull data?
If you want to push, you could write microbatches to salesforce (if possible of course).
For pull, you might wanna use the SQL api of databricks sql, OR use some kind of pub/sub tool like kafka etc where all changes are published, which can be fetched later on.

So perhaps some more info on 'Salesforce'. Is it salesforce cloud or the database?

Re: Expose delta table data to Salesforce - odata?

Ruby8376 — Mon, 25 Mar 2024 17:06:15 GMT

@-werners- we don't want to persist data on SF cloud but just show it on the ui. It will be pulled on demand from salesforceside using external objects. is odata good option?

Re: Expose delta table data to Salesforce - odata?

-werners- — Tue, 26 Mar 2024 12:19:30 GMT

I would not look at OData, as Databricks itself has no out-of-the-box OData integration. So this means you would have to do this yourself.
Instead I would look at the databricks-sql sql api. So the UI you are talking about is doing a REST call to the query api and databricks sql sends the data.
Something like that?

Re: Expose delta table data to Salesforce - odata?

Ruby8376 — Tue, 26 Mar 2024 14:18:11 GMT

even for this case, there would be some custom development. Salesforce doesn't natively support sending SQL queries directly to external databases or APIs. therefore, either we would need middleware between salesforce and databricks or develope custom apex code on salesforce(to authenticate, request, process repsonse, etc).

Re: Expose delta table data to Salesforce - odata?

-werners- — Tue, 26 Mar 2024 14:21:21 GMT

I see, is there a possibility in SF to define an external location/datasource?
Just guessing here, as these type of packages are really good in isolating data, not integrating it.

Re: Expose delta table data to Salesforce - odata?

Ruby8376 — Fri, 19 Apr 2024 13:46:09 GMT

Hi @werners

So, we have decided on this flow: Salesforce connect <->APIM <->Webapp(Odata)<->databricks sql API <- Delta tables

2 questions here:

1. can you share link/documentation on how we can integrate databricks <-delta tables? do we need serverless sql warehouse for sure or can we just use a databricks sql endpoint for querying??

2. For authentication, we have chosen Oauth machineto machine authentication as per microsoft recommendation. in this process do we use external table to share the same delta table? OAuth machine-to-machine (M2M) authentication | Databricks on AWS

@-werners- Please help

Re: Expose delta table data to Salesforce - odata?

fegvilela — Thu, 21 Nov 2024 14:24:18 GMT

Hey, I think this might help

https://www.salesforce.com/uk/news/press-releases/2024/04/25/zero-copy-partner-network/