cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

Delta Tables

bgarcia
New Contributor III

Iโ€™m beginning my journey into Delta Tables and one thing that is still confusing me is where is the best place to save your delta tables if you need to query them later.

For example I'm migrating several tables from on-prem to azure databricks into individual delta tables. My question is, should I save the individual delta tables which could be significant in size into the DBFS databricks internal storage, or should I mount a blob storage location and save the delta lake tables there? What do people normally do in these situations?

1 ACCEPTED SOLUTION

Accepted Solutions

fshimamoto
New Contributor III

I usually recommend people to store data in a separate storage account (either mounted, or used directly), and don't use the internal storage of workspace for that tasks. Primary reason - it's easier to share this data with other workspaces, or other systems if it's necessary. Internal storage should be primarily used for temp files, libraries, init scripts, etc.

There is a number of useful guides available that can help:

https://github.com/Azure/AzureDatabricksBestPractices/blob/master/toc.md

โ€‹

https://github.com/Azure/AzureDatabricksBestPractices/blob/master/toc.md#Do-not-Store-any-Production...

โ€‹

https://github.com/hurtn/datalake-ADLS-access-patterns-with-Databricks

View solution in original post

1 REPLY 1

fshimamoto
New Contributor III

I usually recommend people to store data in a separate storage account (either mounted, or used directly), and don't use the internal storage of workspace for that tasks. Primary reason - it's easier to share this data with other workspaces, or other systems if it's necessary. Internal storage should be primarily used for temp files, libraries, init scripts, etc.

There is a number of useful guides available that can help:

https://github.com/Azure/AzureDatabricksBestPractices/blob/master/toc.md

โ€‹

https://github.com/Azure/AzureDatabricksBestPractices/blob/master/toc.md#Do-not-Store-any-Production...

โ€‹

https://github.com/hurtn/datalake-ADLS-access-patterns-with-Databricks

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโ€™t want to miss the chance to attend and share knowledge.

If there isnโ€™t a group near you, start one and help create a community that brings people together.

Request a New Group