Increase stack size Databricks
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-03-2024 08:56 AM - edited 06-03-2024 10:34 AM
Hi everyone
I'm currently running a shell script in a notebook, and I'm encountering a segmentation fault. This is due to the stack size limitation. I'd like to increase the stack size using ulimit -s unlimited, but I'm facing issues with setting this limit in the notebook environment.
I am using:
2-12 Workers 256-1,536 GB Memory 64-384 Cores
1 Driver256 GB Memory, 64 Cores
Runtime15.2.x-scala2.12
Could anyone provide guidance on how to properly increase the stack size for my shell script using Notebooks in Databricks? Any tips or alternative solutions to avoid the segmentation fault would also be greatly appreciated.