cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

Real time Streaming to AI Engine at Databricks

abhietc31
Visitor

Hi there,

Usercase: Fraud Detection 

Technology : Databricks, Kafka, Microsoft Foundry, Dashboard showing real-time reports

Ask: Please let me know if any reference or technology details to achieve 1-2 milli second (Max) latency using above technology stack. 

Thanks,

Abhishek

2 REPLIES 2

DoTA
Valued Contributor III

One thing worth pinning down first is where the 1-2 ms is measured. If it is the whole path from the transaction event to a fraud score back to the payment service, that budget is tighter than a Kafka hop plus a network round trip to a hosted model endpoint, because the round trip alone tends to eat it. If it is only the streaming leg (Kafka in, a feature or rule step, Kafka out), that is a different conversation.

On the Databricks side, the closest thing to what you describe is real-time mode in Structured Streaming. The docs show sample end-to-end p99 around 4 to 5 ms for simple pipelines reading and writing Kafka, and they say themselves that the number depends on your business logic and has to be measured with it. It needs a dedicated classic cluster on a recent runtime with the config flag turned on, it was public preview per the docs, and foreachBatch is not supported. So even the good case is a few ms, not 1-2.

If 1-2 ms is a hard requirement at the decision point, the usual pattern is to keep scoring inside or right next to the transaction service (small model or rules held in memory, features cached) and use Databricks for training, feature computation, backtesting and the near-real-time dashboard. I would benchmark with your real payload and a p99 target before committing. The e2eLatencyMs metric in the streaming progress is a decent first measurement.

saurabh18cs
Honored Contributor III

Hi @abhietc31 I assume this is what you are looking for :
https://docs.databricks.com/aws/en/structured-streaming/real-time/concepts

Possible in both pyspark structured streaming pipelines or lakeflow pipelines. Depends how you are curating your pipelines.

Real-time mode (RTM) is a trigger type for Structured Streaming that is up to 100x faster than micro-batch mode. Use RTM for operational workloads that require immediate response to streaming data, such as fraud detection and real-time personalization.

 

e.g.

query = (
spark.readStream
.format("kafka")
.option("kafka.bootstrap.servers", broker_address)
.option("startingOffsets", "earliest")
.option("subscribe", input_topic)
.load()
.writeStream
.format("kafka")
.option("kafka.bootstrap.servers", broker_address)
.option("topic", output_topic)
.option("checkpointLocation", checkpoint_location)
.trigger(realTime="5 minutes")
.outputMode("update")
.start()
)