topic Re: Running a cell with R-script keeps waiting status in Data Engineering

Running a cell with R-script keeps waiting status

sticky — Tue, 01 Oct 2024 12:28:56 GMT

So, i have a R-notebook with different cells and a '15.4 LTS ML (includes Apache Spark 3.5.0, Scala 2.12)' cluster.

If i select 'run all' all cells will be run immediately and the run finishes quickly and fine. But if i would like to run the cells one by one, the cells often wait for a very long time (half an houre or an houre) to run or it just will not run.

I checked some logs on the cluster but i could not find anything why it's waiting. i am the only user of the cluster.

If i restart the cluster i can run all the cells at once again and it goes fine.

How can i solve this and how can i check what is happening on the cluster and why it is waiting to run my cell?

Re: Running a cell with R-script keeps waiting status

sticky — Fri, 04 Oct 2024 07:43:50 GMT

Update: i used different types of clusters (single node, other drivers) with the same result. When i was checking my R-code by running cell by cell and line by line (in the responsible cell for this problem). it turns out that the glm-functions of the standard stats-package causes this problem. This is the most important function i use in this notebook. Did anyone else have this problem with the glm-function?

Re: Running a cell with R-script keeps waiting status

sticky — Fri, 04 Oct 2024 15:03:52 GMT

Today, I tried the glm function from the SparkR package. And it seemed to have initially solved the problem with the glm function. However, when you save the result of the glm function in a variable, things seem to go wrong. But only when the variable explorer is open. When it refreshes, things go wrong, and you can't run anything anymore. If you keep the variable explorer closed, there seems to be no issue. So, I suspect that the variable explorer has trouble loading the glm object or something, and everything hangs on that. This object is generally quite large.