fshimamoto
New Contributor III

I usually recommend people to store data in a separate storage account (either mounted, or used directly), and don't use the internal storage of workspace for that tasks. Primary reason - it's easier to share this data with other workspaces, or other systems if it's necessary. Internal storage should be primarily used for temp files, libraries, init scripts, etc.

There is a number of useful guides available that can help:

https://github.com/Azure/AzureDatabricksBestPractices/blob/master/toc.md

https://github.com/Azure/AzureDatabricksBestPractices/blob/master/toc.md#Do-not-Store-any-Production...

https://github.com/hurtn/datalake-ADLS-access-patterns-with-Databricks

View solution in original post