It's important to keep in mind that there are 2 file systems:

1. The file system on the local machines that are part of the cluster
2. The distributed file system https://docs.databricks.com/data/databricks-file-system.html

When you use python w/out spark such as with sklearn, its only on the driver and local is local on the driver. That will go away when the cluster does.

Try %sh ls / and %fs ls and see the differences