Ok, the nc-file is of a file type not known to spark.
So that is why you read it in python, to convert it, I suppose.
Thing is that if you use plain python, Databricks behaves in the same way as your local computer with the same limits.
The power of spark is the parallel processing. But spark does not know this file format so here we are.
I do not know this nc-file format, but what you need is a reader which will work on Spark
(pyspark if you use python).