Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-30-2021 04:23 PM
I don't. This is moreso a hypothetical question. I do have a script where I'm using pandas, but I want to account for my dataset growing larger over time. If my runtime ends up taking minutes... then would I benefit from using pyspark instead? And if yes, can I just replace the high workload intensive tasks and keep everything else in pandas.... note: this is my first time in databricks so some of my assumptions may be off here. Please feel free to correct me if I'm wrong!