topic Re: Where does custom state store the data in Data Engineering

Where does custom state store the data

Soma — Sun, 09 Apr 2023 11:34:10 GMT

There are couple of custom state functions like mapgroupswithstate,

ApplyinpandaswithState

Which has a internal state maintained is it maintained in same statestore(rocksdb) as aggregation state store function

Re: Where does custom state store the data

Anonymous — Mon, 10 Apr 2023 13:03:09 GMT

@somanath Sankaran :

Yes, custom state functions like mapGroupsWithState and applyInPandasWithState use the same state store as the built-in aggregation state store. By default, this state is stored in RocksDB, which is an embedded, persistent key-value store that is optimized for storing and retrieving large amounts of data.

The state store is managed by the Databricks runtime and is automatically distributed across the worker nodes in the cluster. This allows the state to be shared and updated across multiple tasks running in parallel. The state is also fault-tolerant and can be recovered in case of a node failure.

When using custom state functions, it's important to keep in mind that the amount of state maintained by the function can have a significant impact on cluster performance and memory usage. It's important to properly configure the state timeout and eviction policies to ensure that old, unused state is regularly cleaned up to avoid running out of memory.

Re: Where does custom state store the data

Anonymous — Wed, 12 Apr 2023 07:34:49 GMT

Hi @somanath Sankaran

Thank you for posting your question in our community! We are happy to assist you.

To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers your question?

This will also help other community members who may have similar questions in the future. Thank you for your participation and let us know if you need any further assistance!

Re: Where does custom state store the data

PushkarDeole — Fri, 19 Jul 2024 05:52:14 GMT

Hello,

As I understand from above response, the state store is distributed across worker nodes, which means the state store would be stored on the local storage of each worker node. Correct me I understood incorrectly?

So, if it is stored on local storage of worker node, then that storage is ephemeral storage which means it would be wiped out on restart of that worker node. In this case how the state is restored on restart or failure of a worker node?