HI @akdav ,

Ok, I've recreated your scenario (more or less). So I enabled SFTP on my storage account and created home directory for my SFTP user:

Then in databricks I've enabled file events for an external location (which is recommended). To enable it, you need to make sure that your Unity Catalog Access Connector has appropriate permissions. So check if your managed identity has following roles:

• Storage Account Contributor
• Storage Blob Data Contributor
• EventGrid EventSubscription Contributor
• Storage Queue Data Contributor: Required only if you want Azure Databricks to create a subscription and events in Azure Data Lake Storage for you. If you do not enable this role, you must create the Azure storage queue yourself).

Manage external locations - Azure Databricks | Microsoft Learn

Next, you need to enable file events for your external location. Go to Unity Catalog and click external locations.

Select the one for which you want to enable file event.
Now, once you are inside external location - click edit button:

Then tick Enable file events and click Auto-fill access connector ID

Now you can configure you job with file arrival trigger. If everything went smoothly you should see event grid system topic and event subscription created for you by Databricks in your storage account

 I've tested it by uploading some file using WinSCP and file arrival trigger worked like a charm 🙂