@ShresthaBaburam 

We inquired about this a few days ago and checked with Databricks. They were working on the issue, but no ETA was provided. You can find more details here: Databricks Community Link.

However, to address this use case, we followed the steps below:

1. Configure Autoloader with Directory Listing. [ P:S:- use trigger(availableNow=True) ]
2. Capture the File Path: Use the _metadata column to capture the file path of the newly arrived file:
   df_with_path = df.withColumn("input_file_path", input_file_name())
3. Pass the File Path to the Next Task: Once the file path is captured, pass it to the next task in the pipeline using the appropriate workflow or task parameter mechanism.

I hope this helps.