topic Trigger.AvailableOnce in Pyspark? in Data Engineering

Trigger.AvailableOnce in Pyspark?

GMO — Fri, 19 Nov 2021 15:16:06 GMT

There’s a new Trigger.AvailableOnce option in runtime 10.1 that we need to process a large folder bit by bit using Autoloader. But I don’t see how to engage this from pyspark. Is this accessible from scala only or is it available in pyspark? Thanks.

Re: Trigger.AvailableOnce in Pyspark?

GMO — Fri, 19 Nov 2021 15:32:40 GMT

Confirmed this is currently only available in Scala (as of Nov 2021), pyspark support forthcoming in a future release.

Re: Trigger.AvailableOnce in Pyspark?

Anonymous — Fri, 19 Nov 2021 23:42:56 GMT

@Mark Murphy - Thanks for that information.

Re: Trigger.AvailableOnce in Pyspark?

pottsork — Wed, 16 Feb 2022 15:51:08 GMT

Any update on this issue? I can see that one can use .trigger(availableNow=True) i DBR 10.3 (On Azure Databricks).... Unfortunately I can't get it to work with Autoloader. Is this supported?

Additionally, can't find any answers when skimming through the release notes.

Re: Trigger.AvailableOnce in Pyspark?

drewster — Tue, 22 Feb 2022 23:17:34 GMT

I got it working with cloudFiles (autoloader) source except my autoloader is not subscribing to the ADLSgen2 storage.Just using cloudFiles like a normal source.