cancel
Showing results for 
Search instead for 
Did you mean: 
Machine Learning
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

User16752245312
by New Contributor III
  • 2403 Views
  • 5 replies
  • 2 kudos

When running structured streaming jobs in production, what are the general best practices to reduce cost?

Consider a basic structured streaming use case of aggregating the data, perform some basic data cleaning transformation, and merge into a historical aggregate dataset.

  • 2403 Views
  • 5 replies
  • 2 kudos
Latest Reply
lawrence009
Contributor
  • 2 kudos

I second the recommendations: auto load with trigger, batch processing instead of continuous streaming where use case permits. In addition, test with a small batch firstfavor fewer larger workers over more smaller workersadjust your job cluster over...

  • 2 kudos
4 More Replies
lawrence009
by Contributor
  • 713 Views
  • 2 replies
  • 3 kudos

Streaming Source for Feature Store (and outputMode)

To save computing resource and time, can I use streaming source in a batch mode (similar to Auto Loader) to update my feature store as my source table receives row update or is appended with new rows?

  • 713 Views
  • 2 replies
  • 3 kudos
Latest Reply
Meghala
Valued Contributor II
  • 3 kudos

yes you can schedule the job to process the data with auto loader

  • 3 kudos
1 More Replies
Labels