Streaming IoT Hub Data Using Delta Live Table Pipeline

dfish8124
Databricks Partner

Hello,

I'm attempting to stream IoT Hub data using a delta live table pipeline.  The issue I'm running into is that eventhub streaming isn't supported on shared clusters ([UNSUPPORTED_STREAMING_SOURCE_PERMISSION_ENFORCED] Data source eventhubs is not supported as a streaming source on a shared cluster). 

I've tried to configure the delta live table pipeline to use a single user cluster using "data_security_mode" : "SINGLE_USER" but I get this error ([DLT ERROR CODE: INVALID_CLUSTER_SETTING.CLIENT_ERROR] Cluster attribute `data_security_mode' for cluster 'default' is not allowed. Please remove `data_security_mode' from cluster 'default' in the pipeline settings.) Looking at the documentation it seems that option is not user settable.

How can I run a delta live table pipeline with a single user cluster?  And if I can't do that how can I stream IoT Hub data to a table in the unity catalog?

Thanks

szymon_dybczak
Esteemed Contributor III

Hi @dfish8124 ,

Is is possible to share with us code?