topic Re: Can Delta Lake completely host a data warehouse and replace Redshift? in Data Engineering

Can Delta Lake completely host a data warehouse and replace Redshift?

Sweta — Thu, 13 Oct 2022 03:50:35 GMT

Our use case is simple - to store our PB scale data and transform and use for BI, reporting and analytics. As my title says am trying to eliminate expenditure on Redshift as we are starting as a green field.

I know I have designed/used just Delta lake as an analytics data warehouse though we still used Redshift as a legacy warehouse by other teams in my previous org. But just trying to get some consensus here to gain more confidence.

Thanks,

Swetha

Re: Can Delta Lake completely host a data warehouse and replace Redshift?

PriyaAnanthram — Thu, 13 Oct 2022 06:01:41 GMT

Yeah I believe so delta lake is just getting better and better I think its a great way forward to support usecases similar to yours

Re: Can Delta Lake completely host a data warehouse and replace Redshift?

-werners- — Thu, 13 Oct 2022 08:36:23 GMT

I think it can. If we were a few years earlier I would say: no. But now with databricks sql (or other software like Trino etc) and delta lake, I don't see why it would not work.

As we speak I am looking into getting rid of your Azure Synapse database, and I think we can pull that of without issues.

The main attention point however is to not forget to look at what tools are being used to consume data.

Because from a technical point of view, this is quickly overlooked. f.e. if your company uses some paginated reporting software, you better make sure that will still work.

Re: Can Delta Lake completely host a data warehouse and replace Redshift?

Hubert-Dudek — Tue, 18 Oct 2022 10:57:16 GMT

Yes yes yes 🙂

Re: Can Delta Lake completely host a data warehouse and replace Redshift?

Hubert-Dudek — Tue, 18 Oct 2022 11:01:04 GMT

Yes that data lake/lakehouse is excellent for replacing Redshift but I would not write more details as it is a long topic.

Re: Can Delta Lake completely host a data warehouse and replace Redshift?

Hubert-Dudek — Tue, 18 Oct 2022 11:03:34 GMT

That one wasn't so great, hehe Thank you!

Re: Can Delta Lake completely host a data warehouse and replace Redshift?

Anonymous — Sun, 27 Nov 2022 12:51:29 GMT

Hi @Swetha Marakani

Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help.

We'd love to hear from you.

Thanks!