@Jonathan_ Good that you have given some extra information. Based on that I think that there might be memory issue, since it is a single node cluster, both driver and executor resides in the same machine. 

It would be better if you could tell me the RAM that is allocated.  

If your RAM is around 16gb then you can take a look at PART A and PART B.

You have 16 cores, out of which 15 will be utilized as executors and 1 for driver.

PART A of memory issue:

Check in spark ui, what is the storage memory which is allocated under executor tab, and by default spark memory storageFraction is 0.5. Which means 50% is utilized for storing (cache/persist ) and 50% for execution ( join, groupby ).

For example : if the storage memory is 1 gb, then the execution memory is 1gb. So, if you do persist/cache of df you can see the storage memory getting utilized. 

Based on that if you feel that your storage isnt getting utilized much try to adjust the storageFraction and give more memory for execution.

PART B of memory issue:

Now even the driver needs memory for scheduling tasks, to store the results from executors, monitoring.

You could have had OOM error if you tried to return more data than spark.driver.maxResultSize by default it is 1gb.

These are something to keep in mind.

REG AQE, if you are using spark 3.0 it is available by default.

"a simple count/display", if you are new to spark, count and display are action statements. When spark encounters such action statements, it reads the data and runs all your transformations. 

For example: i have 2 action statement, one count and one action, what happens internally is, for count the data is read then processed and then your result is sent to driver, and it is visible to us. And when display runs, it again reads the data and does the processing and then it is sent to driver and shows your result. So there will be 2 jobs. 

One other issue can be small files issue during read, if your input data has lots of files or small files, there will be more time spent in IO, trying to read each file. If that is the case you can make these into a bigger chunk of file.

"Often in our projects we need to used library that works mainly with data in memory", also a remainder, if any process is running out side spark framework, it is running on driver only. And in local mode driver is 1 core and all the process is running on that one core. 

Hope it helps. 🙂