Input File Path from Autoloader in Delta Live Tables

Enzo_Bahrami
New Contributor III

‎05-25-2023 05:53 PM

Hello everyone!

I was wondering if there is any way to get the subdirectories in which the file resides while loading while loading using Autoloader with DLT. For example:

def customer():

 return (

  spark.readStream.format('cloudfiles')

    .option('cloudFiles.format', 'json')

.load(f'/Users/customer/dynamic_dlt/raw/customer/customer_*.json')

.withColumn('input_file', input_file_name())

But instead of input_file_name() something like input_file_path()

0 Kudos

pvignesh92
Honored Contributor

‎05-26-2023 12:24 AM

Hi @Parsa Bahraminejad​ , I'm not sure of an inbuilt function to get the sub directory name but you can easily do this by using split function on '/' on the input_file_name() as the file name will have the complete path as well.

Please let me know if this helps.

View solution in original post

0 Kudos

Anonymous
Not applicable

‎06-01-2023 01:37 AM

Hi @Parsa Bahraminejad​ 

We haven't heard from you since the last response from @Vigneshraja Palaniraj​ ​, and I was checking back to see if her suggestions helped you.

Or else, If you have any solution, please share it with the community, as it can be helpful to others. 

Also, Please don't forget to click on the "Select As Best" button whenever the information provided helps resolve your question.

0 Kudos