Streaming Source for Feature Store (and outputMode)
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-20-2022 02:05 AM
To save computing resource and time, can I use streaming source in a batch mode (similar to Auto Loader) to update my feature store as my source table receives row update or is appended with new rows?
Labels:
- Labels:
-
Feature Store
-
Structured streaming
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-26-2022 01:32 AM
Yes you can. Use Streaming doesn't mean having a cluster running all the time. You can schedule your job to run on the top of every hour or everyday to process the data with Autoloader.
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-26-2022 02:59 AM
yes you can schedule the job to process the data with auto loader
