cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

How to get the files with a prefix in Pyspark from s3 bucket?

User16826994223
Honored Contributor III

I have different files in my s3. Now I want to get the files which starts with cop_

1 REPLY 1

User16826994223
Honored Contributor III

You are referencing a FileInfo object when calling 

.startswith()

and not a string.

The filename is a property of the 

FileInfo

 object, so this should work

filename.name.startswith('cop_ ')

 should work.

Welcome to Databricks Community: Lets learn, network and celebrate together

Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections. 

Click here to register and join today! 

Engage in exciting technical discussions, join a group with your peers and meet our Featured Members.