cancel
Showing results for 
Search instead for 
Did you mean: 
Community Discussions
Connect with fellow community members to discuss general topics related to the Databricks platform, industry trends, and best practices. Share experiences, ask questions, and foster collaboration within the community.
cancel
Showing results for 
Search instead for 
Did you mean: 

INVALID_PARAMETER_VALUE: There are more than 1001 files. Please specify [LIMIT] to limit the num....

databricksdummy
New Contributor

I am running Azure Databricks DBX Runtime 13.3 and I am seeing the following error in my workspace:

Last execution failed

# doesn't work in DBX Runtime 13.3 and above
# limitation is it can't read more than 1001 files

dbutils.fs.ls("[path]")

AnalysisException: INVALID_PARAMETER_VALUE: There are more than 1001 files. Please specify [LIMIT] to limit the number of files returned.

 

I know that the error is pretty clearly stated as to why this exception occurred, but I was tasked with finding documentation on this limitation, but have not been able to find any. Would someone be able to point me in the appropriate direction for this documentation or pointed in the direction of a communication around this topic/ETA?

Thanks!

1 REPLY 1

Kaniz_Fatma
Community Manager
Community Manager

Hi @databricksdummyIn Databricks Runtime 13.3 LTS, powered by Apache Spark™ 3.4.1, there is a restriction related to listing files using dbutils.fs.ls. Specifically, it cannot read more than 1001 files at once. When you encounter this error, you’ll need to specify a limit to control the number of files returned.

This is a known limitation and is not documented in the official Databricks Runtime 13.3 LTS release notes.

Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!