Azure Databricks to Adobe Experience Platform

[tonylax6]

I'm using Azure databricks and am attempting to stream near real-time data from databricks into the Adobe Experience Platform to ingest into the AEP schema for profile enrichment.

We are running into an issue with the API and streaming, so we are currently connecting the platforms through SFTP . Has anyone created the connection between these two platforms in a streaming fashion to support more near real-time updates into the Adobe Experience Platform schema architecture?

[tsilverstrim]

Hi Tony... there are several ways to accomplish this based on the non-functional requirements of your Use Case.  What does near real-time mean from a standpoint of signal to activation time from when the data is present in Databricks to when the activation via, for example, Streaming Destination should occur (not to get to deep in the weeds, but also consider the SLT... are you looking for 75%, 95%, 99% within the target time).  This will drive the design and the short answer is that converting Materialized data in a Database into a Stream and vice versa would occur upstream of AEP and suggest looking at https://docs.databricks.com/aws/en/dlt/what-is-change-data-capture , One suggestion is to also implement a full sync capability... issues can emerge and having a "get to a known data state" capability is prudent for high value/critical data and can dramatically reduce outage duration.