cancel
Showing results for 
Search instead for 
Did you mean: 
Machine Learning
Dive into the world of machine learning on the Databricks platform. Explore discussions on algorithms, model training, deployment, and more. Connect with ML enthusiasts and experts.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

User16752245312
by New Contributor III
  • 2453 Views
  • 5 replies
  • 2 kudos

When running structured streaming jobs in production, what are the general best practices to reduce cost?

Consider a basic structured streaming use case of aggregating the data, perform some basic data cleaning transformation, and merge into a historical aggregate dataset.

  • 2453 Views
  • 5 replies
  • 2 kudos
Latest Reply
lawrence009
Contributor
  • 2 kudos

I second the recommendations: auto load with trigger, batch processing instead of continuous streaming where use case permits. In addition, test with a small batch firstfavor fewer larger workers over more smaller workersadjust your job cluster over...

  • 2 kudos
4 More Replies
Joseph_B
by New Contributor III
  • 683 Views
  • 0 replies
  • 1 kudos

mlflow.org

2021-09 webinar: Automating the ML Lifecycle With Databricks Machine Learning (Post 2 of 2)Thank you to everyone who joined! You can access the on-demand recording here and the code in this Github repo.We're sharing a subset of the questions asked an...

  • 683 Views
  • 0 replies
  • 1 kudos
Labels