Delta Tables

[bgarcia]

I’m beginning my journey into Delta Tables and one thing that is still confusing me is where is the best place to save your delta tables if you need to query them later.

For example I'm migrating several tables from on-prem to azure databricks into individual delta tables. My question is, should I save the individual delta tables which could be significant in size into the DBFS databricks internal storage, or should I mount a blob storage location and save the delta lake tables there? What do people normally do in these situations?

[fshimamoto · Accepted solution]

I usually recommend people to store data in a separate storage account (either mounted, or used directly), and don't use the internal storage of workspace for that tasks. Primary reason - it's easier to share this data with other workspaces, or other systems if it's necessary. Internal storage should be primarily used for temp files, libraries, init scripts, etc.

There is a number of useful guides available that can help:

https://github.com/Azure/AzureDatabricksBestPractices/blob/master/toc.md

https://github.com/Azure/AzureDatabricksBestPractices/blob/master/toc.md#Do-not-Store-any-Production...

https://github.com/hurtn/datalake-ADLS-access-patterns-with-Databricks