topic Autoloader file notification mode error using UC in Data Engineering

Autoloader file notification mode error using UC

rimaissa — Tue, 09 Jul 2024 14:34:21 GMT

We have a DLT pipeline we've created that is using autoloader file notification mode. The pipeline ran fine before moving it to UC. Now that we're using UC, we are getting an AWS permissions issue when the autoloader file notification mode is set to true. We've checked all of our AWS permissions and everything is configured correctly. We do not want to parameterize our accesskey or secret keys. Have there been any updates about this?

@dlt.table(

table_properties={

"quality": "bronze",

"pipelines.autoOptimize.managed":"true",

}

)

def bronze():

return (

spark.readStream.format("cloudFiles")

.option("cloudFiles.format", "avro")

.option("cloudFiles.useNotifications", "true")

.option("cloudFiles.backfillInterval", "1 day")

.option("pathGlobFilter", "*.avro")

.option("avroSchema", avro_schema)

.load(data_source)

Re: Autoloader file notification mode error using UC

rimaissa — Fri, 12 Jul 2024 15:17:01 GMT

Hi @Retired_mod, thank you. We are trying to understand if this is an actual issue with using autoloader file notifications in UC since autoloader file notification mode is only supported on single user clusters. Seems like this won't work because of that issue?

Re: Autoloader file notification mode error using UC

cgrant — Thu, 12 Dec 2024 05:30:54 GMT

Please make sure you are using Dedicated (single-user) clusters when authenticating to the file notifications service when attempting to authenticate to SQS via instance profile authentication. This likely will change in the future, so stay posted.