topic Re: File Trigger using azure file share in unity Catalog in Data Engineering

File Trigger using azure file share in unity Catalog

angel_ba — Sun, 10 Mar 2024 12:03:22 GMT

Hello,

I have got the unity catalog eanbled in my workspace. the file srae manually copied by customers in azure file share(domain joint account, wabs) on adhoc basis. I would like to add a file trigger on the job so that as soon as file arrives in the file share it should get copied to my container(abfss) and triggers the job. But I am not able to mount azure file share but only abfss. How can I resolve it?

Re: File Trigger using azure file share in unity Catalog

Diego33 — Thu, 05 Sep 2024 16:29:38 GMT

Hi Kaniz. I have been trying to follow those steps for the past 2 days, multiple work around in python to try to mount SBM on databricks : 3. Mounting Azure File Share (WABS): 5. Copy the provided script. (MSFT Azure script proveded in Powershell or Bash, it does not work on Python Notebook, I tried multiple work around without success, I was not able to use Databricks Notebook to mount the Azure File Share. can you help?

Re: File Trigger using azure file share in unity Catalog

adriennn — Tue, 17 Sep 2024 10:54:58 GMT

@Diego33 Kaniz is half-bot half-human, but unfortunately not gracing us with "sorry for the confusion" responses.

After a quick search, I thought that maybe there's a possiblity to find use the web terminal and do a manual mount with the bash script that Microsoft provides. But all I get is mount:permission denied. I have a use case on my hands with File Share so I thought I'd give it a try.

It's probably easier to work with the Azure Python SDK to access data from azure storage fileshares.

And @angel_ba, there are vague references to a File Share trigger becoming available in Azure Function but it hasn't materialized and is not referenced as trigger, and it certainly isn't available in Databricks. That said, if I had to build a system that must pick up changed files in a File Share, I would use powershell to list the files (maybe have a landing area, to prevent having too much stuff to sift through), then you get the metadata containing the field "LastModified" with the above command, and then you can decide what to do. Or simply move the file from the File Share to blob storage and then it solves all the other issues.