Databricks Community

Dimitry · 09-02-2025

Hi allI got a job trigger by a file event on the external location.The location and jobs triggers are working fine when uploading file via Azure Portal.I need SFTP trigger, so I went into the event grid, found subscription for the storage account on ...

Dimitry · 07-11-2025

Hi allI've followed the book for creating external location for Azure Data Lake Storage (ADLS V2) using account connectorI've granted all required permissions to the connector:I've created a "stock" container on that above mentioned "devtyremeshare" ...

Dimitry · 06-17-2025

Hi all I have 2 clusters, that look identical but one runs my UDF in parallel another one does not.The ones that do is personal, the bad one is shared.import pandas as pd from datetime import datetime from time import sleep import threading # test f...

Dimitry · 06-08-2025

I've got an UDF which I call using applyInPandasThat UDF is to distribute API calls.It uses my custom .py library files that make these calls.Everything worked until I use `dbutils.widgets.get` and `dbutils.secrets.get` inside these libraries.It thro...

Dimitry · 06-08-2025

I've read all relevant articles but none have solution that I could understand. Sorry I'm new to it.I have a simple UDF to demonstrate the problem:df = spark.createDataFrame([(1, 1.0, 'a'), (1, 2.0, 'b'), (2, 3.0, 'c'), (2, 5.0, 'd'), (2, 10.0, 'e')]...

Dimitry · 09-04-2025

Yeah, but in my case I just archive the file as soon as it gets processed, and it will be handful tops. I guess majority of cases are like that and when you need more in one place... perhaps you have a budget to incorporate ADF/Function/Logic app. I ...

Dimitry · 09-04-2025

FYI - look at this Solved: Job File Event Trigger not firing for SftpCommit a... - Databricks Community - 128356You also copied the same solution. But the trick is - you DONT use events. Just turn it off. No queue.In this case File Event trigger will...

Dimitry · 09-04-2025

Oh mate, you just made my day, night and the next few day altogether.Yes, it works! It does!!!

Dimitry · 09-04-2025

Better question, what do you mean under polling? I can have only these triggers:Continuous just runs the job non stop, scheduled on the intervals, and file trigger is that what's not working.So did you use some external crutches to poll?

Dimitry · 09-04-2025

--removed

Databricks Community

User Stats

User Activity

Unreliable file events on Azure Storage (SFTP) for job trigger

Problem activating File Events for External Location / ADLS V2

Struggle to parallelize UDF

UDF fails with "No module named 'dbruntime'" when using dbutils

How to "Python versions in the Spark Connect client and server are different. " in UDF

Re: Unreliable file events on Azure Storage (SFTP) for job trigger

Re: Unreliable file events on Azure Storage (SFTP) for job trigger

Re: Job File Event Trigger not firing for SftpCommit and SftpCreate

Re: Job File Event Trigger not firing for SftpCommit and SftpCreate

Re: Job File Event Trigger not firing for SftpCommit and SftpCreate