topic Re: Input File Path from Autoloader in Delta Live Tables in Data Engineering

Input File Path from Autoloader in Delta Live Tables

Enzo_Bahrami — Fri, 26 May 2023 00:53:30 GMT

Hello everyone!

I was wondering if there is any way to get the subdirectories in which the file resides while loading while loading using Autoloader with DLT. For example:

def customer():

return (

spark.readStream.format('cloudfiles')

.option('cloudFiles.format', 'json')

.load(f'/Users/customer/dynamic_dlt/raw/customer/customer_*.json')

.withColumn('input_file', input_file_name())

But instead of input_file_name() something like input_file_path()

Re: Input File Path from Autoloader in Delta Live Tables

pvignesh92 — Fri, 26 May 2023 07:24:21 GMT

Hi @Parsa Bahraminejad , I'm not sure of an inbuilt function to get the sub directory name but you can easily do this by using split function on '/' on the input_file_name() as the file name will have the complete path as well.

Please let me know if this helps.

Re: Input File Path from Autoloader in Delta Live Tables

Anonymous — Thu, 01 Jun 2023 08:37:22 GMT

Hi @Parsa Bahraminejad

We haven't heard from you since the last response from @Vigneshraja Palaniraj , and I was checking back to see if her suggestions helped you.

Or else, If you have any solution, please share it with the community, as it can be helpful to others.

Also, Please don't forget to click on the "Select As Best" button whenever the information provided helps resolve your question.