Databricks Community

Rik · 08-07-2023

We are using the UC mechanism for triggering jobs on file arrival, as described here: https://learn.microsoft.com/en-us/azure/databricks/workflows/jobs/file-arrival-triggers.Unfortunately, the trigger doesn't actually pass the file-path that is gener...

Rik · 04-06-2023

I have disabled the IP Access List on my workspace and am trying to add an IP list through the IP Access List API. However, when adding a list, I get the INVALID_STATE response.The docs mention this is because:"If the new list would block the calling...

Rik

@edejong, agreed. I don't know how to unapprove the solution (I assume only OP can do that), but I did report it to the moderator.

Rik · 08-08-2023

"Alternately, You can use autoloader as part of the triggered pipeline to get the details of the file that arrived."That doesn't quite fit our requirements unfortunately... Are there any plans on adding this functionality?

Rik · 08-07-2023

@Yelf, this is how private link works. Once you enable private link, your per-workspace URL will resolve to privatelink.workspace-url.azuredatabricks.net,If you are on your private network, you should have a private DNS zone setup that resolves this ...

Rik · 08-07-2023

This solution doesn't answer the question...It seems that no additional parameters are passed to the job for file arrivals as described here (https://learn.microsoft.com/en-us/azure/databricks/workflows/jobs/file-arrival-triggers).Any plans on adding...

Rik · 04-07-2023

"One possible workaround could be to (1) temporarily enable the IP Access List feature, (2) add the necessary IP addresses to the list, and then (3) disable the feature again. This way, you can add the IP addresses you need without blocking the curre...

Databricks Community

User Stats

User Activity

File information is not passed to trigger job on file arrival

Incorrect error when adding an IP access list

Re: Get file information while using "Trigger jobs when new files arrive" https://docs.dat

Re: File information is not passed to trigger job on file arrival

Re: Azure Databricks - Difference between protecting the WEB UI with IP Access list or disabling pu

Re: Get file information while using "Trigger jobs when new files arrive" https://docs.dat

Re: Incorrect error when adding an IP access list