Hello,
I'm attempting to stream IoT Hub data using a delta live table pipeline. The issue I'm running into is that eventhub streaming isn't supported on shared clusters ([UNSUPPORTED_STREAMING_SOURCE_PERMISSION_ENFORCED] Data source eventhubs is not supported as a streaming source on a shared cluster).
I've tried to configure the delta live table pipeline to use a single user cluster using "data_security_mode" : "SINGLE_USER" but I get this error ([DLT ERROR CODE: INVALID_CLUSTER_SETTING.CLIENT_ERROR] Cluster attribute `data_security_mode' for cluster 'default' is not allowed. Please remove `data_security_mode' from cluster 'default' in the pipeline settings.) Looking at the documentation it seems that option is not user settable.
How can I run a delta live table pipeline with a single user cluster? And if I can't do that how can I stream IoT Hub data to a table in the unity catalog?
Thanks
