Increase stack size Databricks

tgen — Mon, 03 Jun 2024 17:34:57 GMT

Hi everyone

I'm currently running a shell script in a notebook, and I'm encountering a segmentation fault. This is due to the stack size limitation. I'd like to increase the stack size using ulimit -s unlimited, but I'm facing issues with setting this limit in the notebook environment.

I am using:

2-12 Workers 256-1,536 GB Memory 64-384 Cores

1 Driver256 GB Memory, 64 Cores

Runtime15.2.x-scala2.12

Could anyone provide guidance on how to properly increase the stack size for my shell script using Notebooks in Databricks? Any tips or alternative solutions to avoid the segmentation fault would also be greatly appreciated.

Re: Increase stack size Databricks

tgen — Thu, 06 Jun 2024 15:45:14 GMT

Hi @Retired_mod ,

Thanks for your response. I tried this and unfortunately I could not get it to work.

When I set spark.databricks.driver.maxReplOutputLength to unlimited in the cluster configurations, I got this error message when running in the Notebook: Failure starting repl. Try detaching and re-attaching the notebook. I tried detaching and re-attaching the cluster and continued to get the same message. Looking into it more, it looks like it has to be set to an integer value. I also tried this on the web terminal and I continued to get the segmentation fault error.

Next, I tried setting spark.databricks.driver.maxReplOutputLength to a very high number (e.g. 500000000) and received the same segmentation fault error when running it in the Notebook and web terminal.

Do you have any other ideas of things I could try?

topic Re: Increase stack size Databricks in Data Engineering

Increase stack size Databricks

Re: Increase stack size Databricks