Streaming IoT Hub Data Using Delta Live Table Pipeline

dfish8124 — Thu, 07 Nov 2024 20:18:46 GMT

Hello,

I'm attempting to stream IoT Hub data using a delta live table pipeline. The issue I'm running into is that eventhub streaming isn't supported on shared clusters ([UNSUPPORTED_STREAMING_SOURCE_PERMISSION_ENFORCED] Data source eventhubs is not supported as a streaming source on a shared cluster).

I've tried to configure the delta live table pipeline to use a single user cluster using "data_security_mode" : "SINGLE_USER" but I get this error ([DLT ERROR CODE: INVALID_CLUSTER_SETTING.CLIENT_ERROR] Cluster attribute `data_security_mode' for cluster 'default' is not allowed. Please remove `data_security_mode' from cluster 'default' in the pipeline settings.) Looking at the documentation it seems that option is not user settable.

How can I run a delta live table pipeline with a single user cluster? And if I can't do that how can I stream IoT Hub data to a table in the unity catalog?

Thanks

Re: Streaming IoT Hub Data Using Delta Live Table Pipeline

szymon_dybczak — Wed, 13 Nov 2024 11:40:47 GMT

Hi @dfish8124 ,

Is is possible to share with us code?

topic Re: Streaming IoT Hub Data Using Delta Live Table Pipeline in Data Engineering

Streaming IoT Hub Data Using Delta Live Table Pipeline

Re: Streaming IoT Hub Data Using Delta Live Table Pipeline