Expose delta table data to Salesforce - odata?

[Ruby8376]

HI Looking for suggestiongs to stream on demand data from databricks delta tables to salesforce.

Is odata a good option? 

[Ruby8376]

@-werners- @WernerS  can u help?

[-werners-]

So do you want Databricks to push data into Salesforce or do you want Salesforce to pull data?
If you want to push, you could write microbatches to salesforce (if possible of course).
For pull, you might wanna use the SQL api of databricks sql, OR use some kind of pub/sub tool like kafka etc where all changes are published, which can be fetched later on.

So perhaps some more info on 'Salesforce'.  Is it salesforce cloud or the database?

[Ruby8376]

@-werners- we don't want to persist data on SF cloud but just show it on the ui. It will be pulled on demand from salesforceside using external objects. is odata good option?

[-werners-]

I would not look at OData, as Databricks itself has no out-of-the-box OData integration. So this means you would have to do this yourself.
Instead I would look at the databricks-sql sql api. So the UI you are talking about is doing a REST call to the query api and databricks sql sends the data.
Something like that?

[Ruby8376]

even for this case, there would be some custom development. Salesforce doesn't natively support sending SQL queries directly to external databases or APIs. therefore, either we would need middleware between salesforce and databricks or develope custom apex code on salesforce(to authenticate, request, process repsonse, etc).

[Ruby8376]

Hi @werners

So, we have decided on this flow: Salesforce connect <->APIM <->Webapp(Odata)<->databricks sql API <- Delta tables

2 questions here:

1. can you share link/documentation on how we can integrate databricks <-delta tables? do we need serverless sql warehouse for sure or can we just use a databricks sql endpoint for querying??

2. For authentication, we have chosen Oauth machineto machine authentication as per microsoft recommendation. in this process do we use external table to share the same delta table? OAuth machine-to-machine (M2M) authentication | Databricks on AWS

@-werners-  Please help

[-werners-]

I see, is there a possibility in SF to define an external location/datasource?
Just guessing here, as these type of packages are really good in isolating data, not integrating it.