Python os.listdir() behavior difference between 15.4LTS and 16.4LTS DBRs

[nopal1]

We found that when using os.listdir() in Databricks notebooks to list files stored in the Workspace (i.e., alongside the notebook, not in DBFS), file extensions were missing in Databricks Runtime 14.3 LTS and 15.4 LTS, but appeared correctly in 16.4 LTS.

For example os.listdir('.') would return the following code for the different runtimes:

>>> os.listdir('.') # DBR 14.3LTS, DBR 15.4LTS
['notebookA', 'fileB', 'notebookc']

>>> os.listdir('.') # DBR 16.4LTS, Serverless compute
['notebookA.sql', 'fileB.py', 'notebookc.ipynb']

 On the code level we can handle this, but I'd like to understand and/or track down this breaking change between the different runtimes.

Can you help me with the reason?

[cgrant · Accepted solution]

This is expected and changed in DBR16.2:

In Databricks Runtime 16.2 and above, notebooks are supported as workspace files. 

[nopal1]

Thank you so much!