INVALID_PARAMETER_VALUE: There are more than 1001 files. Please specify [LIMIT] to limit the num....

[databricksdummy]

I am running Azure Databricks DBX Runtime 13.3 and I am seeing the following error in my workspace:

Last execution failed

# doesn't work in DBX Runtime 13.3 and above
# limitation is it can't read more than 1001 files

dbutils.fs.ls("[path]")

AnalysisException: INVALID_PARAMETER_VALUE: There are more than 1001 files. Please specify [LIMIT] to limit the number of files returned.

I know that the error is pretty clearly stated as to why this exception occurred, but I was tasked with finding documentation on this limitation, but have not been able to find any. Would someone be able to point me in the appropriate direction for this documentation or pointed in the direction of a communication around this topic/ETA?

Thanks!

[Kaniz_Fatma]

Hi @databricksdummy, In Databricks Runtime 13.3 LTS, powered by Apache Spark™ 3.4.1, there is a restriction related to listing files using dbutils.fs.ls. Specifically, it cannot read more than 1001 files at once. When you encounter this error, you’ll need to specify a limit to control the number of files returned.

This is a known limitation and is not documented in the official Databricks Runtime 13.3 LTS release notes.

[Rishabh_Tiwari]

Hi @databricksdummy ,

Thank you for reaching out to our community! We're here to help you. 

To ensure we provide you with the best support, could you please take a moment to review the response and choose the one that best answers your question? Your feedback not only helps us assist you better but also benefits other community members who may have similar questions in the future.

If you found the answer helpful, consider giving it a kudo. If the response fully addresses your question, please mark it as the accepted solution. This will help us close the thread and ensure your question is resolved.

We appreciate your participation and are here to assist you further if you need it!

Thanks,

Rishabh