How can I read all the files in a folder on S3 into several pandas dataframes?

zhaoxuan210 — Thu, 16 Jan 2020 17:12:50 GMT

import pandas as pd

import glob

path = "s3://somewhere/" # use your path

all_files = glob.glob(path + "/*.csv")

print(all_files)

li = []

for filename in all_files:

dfi = pd.read_csv(filename,names =['acct_id', 'SOR_ID'], dtype={'acct_id':str,'SOR_ID':str},header = None )

li.append(dfi)

I can read the file if I read one of them. But the glob is not working here. The all_files will return a empty [], how to get the list of the filenames as an array?

Re: How can I read all the files in a folder on S3 into several pandas dataframes?

shyam_9 — Mon, 27 Jan 2020 06:03:43 GMT

Hi @zhaoxuan210,

Please go through the below answer,

https://stackoverflow.com/questions/52855221/reading-multiple-csv-files-from-s3-bucket-with-boto3

topic Re: How can I read all the files in a folder on S3 into several pandas dataframes? in Data Engineering

How can I read all the files in a folder on S3 into several pandas dataframes?

Re: How can I read all the files in a folder on S3 into several pandas dataframes?