Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
07-28-2024 09:47 PM
If you want to cache the dataframe without having to perform action, You can use SparSQL. Python API is always evaluated as lazy for both cache() and persist(). Instead SparkSQL gives option to specify if you want to evaluate it lazy or eager.
Reference -
https://spark.apache.org/docs/3.0.0-preview/sql-ref-syntax-aux-cache-cache-table.html
Pratik Jain