what are the key Data engineering problems that databricks solve?

[harrisriaz]

what are the problem that databricks address from typical data engineering prespective and comparing with other cloud DE tools.

[-werners-]

Can´t speak for others, but for me it is a combination of things:

• autoscaling
• tuned spark clusters
• extra features like Databricks SQL, MLFlow etc
• frequent updates
• reliability

I probably forget a few things, but what is also a great asset is (and this is mainly Spark related and not only Databricks) that you can use python/scala instead of SQL.

Don´t get me wrong: sql is an excellent tool, but when things get complicated I prefer a GP programming language over SQL.

[Rheiman · Accepted solution]

Annoying things databricks solves

1. Sane Data Movement (Fast Parallelized Compute, Table Versioning and History)
2. Environment Management (spark + delta + java) are installed out-of-the-box
3. Cost and Job Monitoring (Overwatch)

I've only worked with it for 6 months and it's really a platform you can build internal practices upon with little overhead.

[Kaniz_Fatma]

Hi @Harris Riaz​ , We haven’t heard from you on the last response from @Ralph David Lagos​  and @Werner Stinckens​ , and I was checking back to see if their suggestions helped you. Or else, If you have any solution, please share it with the community as it can be helpful to others.

[Kaniz_Fatma]

Hi @Harris Riaz​, I appreciate your attempt to choose the best answer for us. I'm glad you got your query resolved.

@rheiman, Thank you for giving an excellent answer 😀.