cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
cancel
Showing results for 
Search instead for 
Did you mean: 

what are the key Data engineering problems that databricks solve?

harrisriaz
New Contributor

what are the problem that databricks address from typical data engineering prespective and comparing with other cloud DE tools.

1 ACCEPTED SOLUTION

Accepted Solutions

Rheiman
Contributor II

Annoying things databricks solves

  1. Sane Data Movement (Fast Parallelized Compute, Table Versioning and History)
  2. Environment Management (spark + delta + java) are installed out-of-the-box
  3. Cost and Job Monitoring (Overwatch)

I've only worked with it for 6 months and it's really a platform you can build internal practices upon with little overhead.

View solution in original post

4 REPLIES 4

-werners-
Esteemed Contributor III

Can´t speak for others, but for me it is a combination of things:

  • autoscaling
  • tuned spark clusters
  • extra features like Databricks SQL, MLFlow etc
  • frequent updates
  • reliability

I probably forget a few things, but what is also a great asset is (and this is mainly Spark related and not only Databricks) that you can use python/scala instead of SQL.

Don´t get me wrong: sql is an excellent tool, but when things get complicated I prefer a GP programming language over SQL.

Rheiman
Contributor II

Annoying things databricks solves

  1. Sane Data Movement (Fast Parallelized Compute, Table Versioning and History)
  2. Environment Management (spark + delta + java) are installed out-of-the-box
  3. Cost and Job Monitoring (Overwatch)

I've only worked with it for 6 months and it's really a platform you can build internal practices upon with little overhead.

Kaniz
Community Manager
Community Manager

Hi @Harris Riaz​ , We haven’t heard from you on the last response from @Ralph David Lagos​  and @Werner Stinckens​ , and I was checking back to see if their suggestions helped you. Or else, If you have any solution, please share it with the community as it can be helpful to others.

Kaniz
Community Manager
Community Manager

Hi @Harris Riaz​, I appreciate your attempt to choose the best answer for us. I'm glad you got your query resolved.

@rheiman, Thank you for giving an excellent answer 😀.

Welcome to Databricks Community: Lets learn, network and celebrate together

Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections. 

Click here to register and join today! 

Engage in exciting technical discussions, join a group with your peers and meet our Featured Members.