Unreliable file events on Azure Storage (SFTP) for job trigger

Dimitry
Valued Contributor

‎09-02-2025 04:57 PM

Hi all

I got a job trigger by a file event on the external location.

The location and jobs triggers are working fine when uploading file via Azure Portal.

I need SFTP trigger, so I went into the event grid, found subscription for the storage account on question and modified filters by adding SftpCommit and SftpRename (sometimes 3rd party uploads .part file and renames).

 

I'm uploading a sample file manually to SFTP and I see the event trigger in the event grid that is displayed as matched event.

Next second I see a message in the Azure Storage account queue (queue is automatically created by Bricks for this external location). Then the message disappears.

Job... nothing.

 

2nd part of the riddle.

Files arrive ONLY by SFTP.

Job does trigger. But it triggers completely out of sync with the events.

I check events in the storage log, local time zone:

Dimitry_2-1756857231122.png

And here are the job runs:

Dimitry_1-1756857151591.png

They are by hours out of sync with the queue.

The job is configured to be queued with 1 concurrent run. A cluster is dedicated to the job.

Anyone seen this before? What is happening?

 

 

0 Kudos