Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
04-30-2025 05:00 PM - edited 04-30-2025 05:02 PM
Hello Hung,
Working with streaming tables is always a challenge. Let's remember we are working with unbounded data so it's important to consider a few points:
- If you are working with Job, you can define your job cluster for each task. Consider the computer's configuration to be able to handle all these multiple streams, specifically Memory, as it could lead to memory spills into your hard drive. You can leverage Sparks UI to collect information about your operations.
- If you are working with heavy stateful streams, you can consider leveraging RocksDB for your state handling
- If you are running Stream-Stream Joins, consider reviewing the details for each in the Structure Streaming Guide
- Consider leveraging Checkpointing of your streams
- As you mentioned, there maybe some ch allenges handling the session, you may want to start small then increase the number of streams until you find which is one the culprit
- Consider limit input rates (files or bytes),perhaps there is a considerable amount of data stream staggering the operation. Alternative you can consider DLT where this is handled automatically.
- Lastly consider your data layout(partitions) and the structure of your operations - Here is an overall guide to optimize data workloads
I wish I could give you a straight answer but hopefully these provides you with some options to keep looking and working it out.
Thank you