Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
Can we create a file based trigger from sharepoint location for excel files from databricks. So my need is to copy the excel files from sharepoint to external volumes in databricks so can it be done using a trigger that whenever the file drops in sharepoint location a trigger is used to copy files. Also if a trigger cannot be used can we use any python code to do it. If cannot be done currently please let me know that we can only do the copy only schedule trigger.
SharePoint does not natively support the event notifications required for Databricks File Arrival Triggers. You can use below
Azure Logic Apps - Create a workflow with "When a file is created in a folder" SharePoint trigger. The workflow copies the file to an ADLS Gen2 path. Once the file lands in ADLS, Databricks File Arrival Trigger can kick off automatically
Python Flow - You can use Python script to list files in SharePoint, compare them against a processed maintenance flow and download new files to ADLS Gen2 or Volumes. You can load the file using Auto Loader or Databricks File Arrival Trigger Flow.
![Community [Industry BrickTalk #3]: Scaling Video Intelligence Using AI on Databricks](/t5/image/serverpage/image-id/25651iB18C1F3FD2C2DA5A/image-size/large?v=v2&px=999)
