topic Re: Appropriate storage account type for reference data (Azure) in Data Engineering

Appropriate storage account type for reference data (Azure)

EDDatabricks — Thu, 24 Aug 2023 14:12:24 GMT

Hello,

We are using a reference dataset for our Production applications. We would like to create a delta table for this dataset to be used from our applications. Currently, manual updates will occur on this dataset through a script on a weekly basis. On the future, we will create a streaming job from Event Hub to propagate changes in real time through streaming jobs (Spark Streaming or DLT Pipelines). The expected rate of updates is 1000 to 10000 insertions/updates per hour.

Currently, we have an Azure blob storage account that contains all production reference data that is mounted on the corresponding databricks instance. The question is whether we should create an ADLS storage account for this Delta Table or the Blob storage account can handle this architecture. Are there any future plans to drop compatibility between blob storage and Delta?

Kind regards.

Re: Appropriate storage account type for reference data (Azure)

Lakshay — Thu, 24 Aug 2023 18:18:22 GMT

@EDDatabricks

In my experience, I have mostly seen users choose ADLS for a similar use case. So, I believe ADLS is a more popular choice.

I have not seen many users go for blob storage but I am hoping someone who has explored blob storage more could add to my comment.

Re: Appropriate storage account type for reference data (Azure)

-werners- — Fri, 25 Aug 2023 11:49:37 GMT

+1 for ADLS. Hierarchical storage, hot/cold/premium storage, things not possible in blob storage