Greetings,
I see that Delta Live Tables has various real-time connectors such as Kafka, Kinesis, Google's Pub Sub, and so on. I also see that Apache had maintained an mqtt connector to Spark through the 2.x series called Bahir, but dropped it in version 3.x which we are currently using.
My use case updates from each node once a minute a variable data payload from 18 KB to 100 KB 24/7/365. I am focusing on one node which averages 95 KB per minute. Log data can come faster than the 1 minute window. We use AWS and the AWS IoT mqtt.
How do I connect the mqtt to a Delta Live Table? Does it require an intermediary of Kafka/Kinesis?
As a intermediate solution, I have an IoT Rule to write an mqtt subscription directly to S3. I doubt that it will be performant but I do intend to test the Autoloader to DLT via S3 capability since it's right in front of me.
Cheers, Joe
