topic Re: extract latest files from ADLS Gen2 mount point in databricks using pyspark in Data Engineering

extract latest files from ADLS Gen2 mount point in databricks using pyspark

pankaj92 — Sun, 01 Aug 2021 17:26:02 GMT

Hi Team,

I am trying to get the latest files from an ADLS mount point directory. I am not sure how to extract latest files ,Last modified Date using Pyspark from ADLS Gen2 storage account. Please let me know asap. Thanks!

I am looking forward your response.

Re: extract latest files from ADLS Gen2 mount point in databricks using pyspark

pankaj92 — Mon, 02 Aug 2021 12:16:13 GMT

Hi Team,

can you please respond here. Thanks!

Re: extract latest files from ADLS Gen2 mount point in databricks using pyspark

pankaj92 — Tue, 10 Aug 2021 12:17:18 GMT

Hi Team,

Can someone please respond here. If someone has already implemented. Thanks!

Re: extract latest files from ADLS Gen2 mount point in databricks using pyspark

Sha_1890 — Tue, 02 Aug 2022 08:48:38 GMT

I am looking for the same too. Appreciate the speedy response.

Re: extract latest files from ADLS Gen2 mount point in databricks using pyspark

Sha_1890 — Tue, 02 Aug 2022 14:00:57 GMT

Hi @pankaj92 ,

I wrote a Python code to pick a latest file from mnt location ,

import os

path = "/dbfs/mnt/xxxx"

filelist=[]

for file_item in os.listdir(path):

filelist.append(file_item)

file=len(filelist)

print(filelist[file-1])

Thanks