topic Real time Streaming to AI Engine at Databricks in Data Engineering

Real time Streaming to AI Engine at Databricks

abhietc31 — Mon, 02 Nov 2026 06:26:26 GMT

Hi there,

Usercase: Fraud Detection

Technology : Databricks, Kafka, Microsoft Foundry, Dashboard showing real-time reports

Ask: Please let me know if any reference or technology details to achieve 1-2 milli second (Max) latency using above technology stack.

Thanks,

Abhishek

Re: Real time Streaming to AI Engine at Databricks

DoTA — Mon, 02 Nov 2026 12:57:40 GMT

One thing worth pinning down first is where the 1-2 ms is measured. If it is the whole path from the transaction event to a fraud score back to the payment service, that budget is tighter than a Kafka hop plus a network round trip to a hosted model endpoint, because the round trip alone tends to eat it. If it is only the streaming leg (Kafka in, a feature or rule step, Kafka out), that is a different conversation.

On the Databricks side, the closest thing to what you describe is real-time mode in Structured Streaming. The docs show sample end-to-end p99 around 4 to 5 ms for simple pipelines reading and writing Kafka, and they say themselves that the number depends on your business logic and has to be measured with it. It needs a dedicated classic cluster on a recent runtime with the config flag turned on, it was public preview per the docs, and foreachBatch is not supported. So even the good case is a few ms, not 1-2.

If 1-2 ms is a hard requirement at the decision point, the usual pattern is to keep scoring inside or right next to the transaction service (small model or rules held in memory, features cached) and use Databricks for training, feature computation, backtesting and the near-real-time dashboard. I would benchmark with your real payload and a p99 target before committing. The e2eLatencyMs metric in the streaming progress is a decent first measurement.

Re: Real time Streaming to AI Engine at Databricks

saurabh18cs — Mon, 02 Nov 2026 14:25:23 GMT

Hi @abhietc31 I assume this is what you are looking for :
https://docs.databricks.com/aws/en/structured-streaming/real-time/concepts

Possible in both pyspark structured streaming pipelines or lakeflow pipelines. Depends how you are curating your pipelines.

Real-time mode (RTM) is a trigger type for Structured Streaming that is up to 100x faster than micro-batch mode. Use RTM for operational workloads that require immediate response to streaming data, such as fraud detection and real-time personalization.

e.g.

query = (
spark.readStream
.format("kafka")
.option("kafka.bootstrap.servers", broker_address)
.option("startingOffsets", "earliest")
.option("subscribe", input_topic)
.load()
.writeStream
.format("kafka")
.option("kafka.bootstrap.servers", broker_address)
.option("topic", output_topic)
.option("checkpointLocation", checkpoint_location)
.trigger(realTime="5 minutes")
.outputMode("update")
.start()
)