Hi,

This is a broad question without much detail. The first thing I'd want to understand is what is the definition of realtime that is required. People say realtime but often they can mean a variety of things:

• What is the latency requirement and where is it coming from and too? is it a stream based feed where you need to process the data and then feedback to another stream? Or do you need to write to structured source such as Lakebase or delta format?
• When we talk about latency do you mean sub second, sub 15 seconds, under a minute or some other variety? This will set whether you can do micro batching or need a true streaming pipeline. Micro batching is usually cheaper and simpler, plus cope better with traffic spikes. If its sub second then you're going to be needing to read from Kafka or event hubs, processing with spark RTM and then writing back to another stream. If its less than a minute, spark declarative pipelines with streaming mode can help you handle this and will cope well with spikes if configured correctly.
• You also need to think about what level of transformation you need to do, this will greatly impact latency.

Some doc links to help with this:

https://docs.databricks.com/aws/en/structured-streaming/real-time/concepts

https://community.databricks.com/t5/technical-blog/a-practitioner-s-guide-to-real-time-mode-on-spark...

https://docs.databricks.com/aws/en/structured-streaming/examples

If you want to provide some more info, happy to help further.

Thanks,

Emma