Serverless Compute no support for Caching data frames

[Dave1967]

Can anyone please tell me why df.cache() and df.persist() are not supported in Serevrless compute?

Many Thanks

[gchandra · Accepted solution]

Global caching functionality (and other global states used on classic clusters) is conceptually hard to represent on serverless computing.

Serverless spark cluster optimizes the cache than the user.

[Dave1967]

Many Thanks

[mrroger]

Hi. I'm not fully convinced that Serverless can optimize Spark cache better than the user, since I still see query plans with recomputed operations. What is the recommended best practice to avoid recomputation in a Serverless environment? Write out intermediate dataframes?

[kunalmishra9]

What I do wish was possible was for serverless to warn that caching is not supported, but not error on a call. It makes switching between compute (serverless & all purpose) brittle and prevents code from easily being interoperable, no matter the compute type, which is significant friction against adopting serverless completely. Even having a parameter (i.e. .cache(try=True) ), would be nice to support this kind of workflow more elegantly.