topic Re: what are the key Data engineering problems that databricks solve? in Data Engineering

what are the key Data engineering problems that databricks solve?

harrisriaz — Wed, 29 Jun 2022 05:26:42 GMT

what are the problem that databricks address from typical data engineering prespective and comparing with other cloud DE tools.

Re: what are the key Data engineering problems that databricks solve?

-werners- — Wed, 29 Jun 2022 07:42:26 GMT

Can´t speak for others, but for me it is a combination of things:

autoscaling
tuned spark clusters
extra features like Databricks SQL, MLFlow etc
frequent updates
reliability

I probably forget a few things, but what is also a great asset is (and this is mainly Spark related and not only Databricks) that you can use python/scala instead of SQL.

Don´t get me wrong: sql is an excellent tool, but when things get complicated I prefer a GP programming language over SQL.

Re: what are the key Data engineering problems that databricks solve?

Rheiman — Thu, 30 Jun 2022 05:44:28 GMT

Annoying things databricks solves

Sane Data Movement (Fast Parallelized Compute, Table Versioning and History)
Environment Management (spark + delta + java) are installed out-of-the-box
Cost and Job Monitoring (Overwatch)

I've only worked with it for 6 months and it's really a platform you can build internal practices upon with little overhead.