Autoloader file notification mode error using UC

[rimaissa]

We have a DLT pipeline we've created that is using autoloader file notification mode. The pipeline ran fine before moving it to UC. Now that we're using UC, we are getting an AWS permissions issue when the autoloader file notification mode is set to true. We've checked all of our AWS permissions and everything is configured correctly. We do not want to parameterize our accesskey or secret keys. Have there been any updates about this? 

@dlt.table(

table_properties={

"quality": "bronze",

"pipelines.autoOptimize.managed":"true",

}

)

def bronze():

return (

spark.readStream.format("cloudFiles")

.option("cloudFiles.format", "avro")

.option("cloudFiles.useNotifications", "true")

.option("cloudFiles.backfillInterval", "1 day")

.option("pathGlobFilter", "*.avro")

.option("avroSchema", avro_schema) 

.load(data_source)

[rimaissa]

Hi @Retired_mod, thank you. We are trying to understand if this is an actual issue with using autoloader file notifications in UC since autoloader file notification mode is only supported on single user clusters. Seems like this won't work because of that issue? 

[cgrant]

Please make sure you are using Dedicated (single-user) clusters when authenticating to the file notifications service when attempting to authenticate to SQS via instance profile authentication. This likely will change in the future, so stay posted.