Trigger.AvailableOnce in Pyspark?

[GMO]

There’s a new Trigger.AvailableOnce option in runtime 10.1 that we need to process a large folder bit by bit using Autoloader. But I don’t see how to engage this from pyspark.  Is this accessible from scala only or is it available in pyspark? Thanks.

[GMO · Accepted solution]

Confirmed this is currently only available in Scala (as of Nov 2021), pyspark support forthcoming in a future release.

[Anonymous]

@Mark Murphy​ - Thanks for that information.

[pottsork]

Any update on this issue? I can see that one can use .trigger(availableNow=True) i DBR 10.3 (On Azure Databricks).... Unfortunately I can't get it to work with Autoloader. Is this supported?

Additionally, can't find any answers when skimming through the release notes.

[drewster]

I got it working with cloudFiles (autoloader) source except my autoloader is not subscribing to the ADLSgen2 storage.Just using cloudFiles like a normal source.