Databricks Community

Also interested in increasing shared memory for use with ray

According to the docs of spark.read.csv(...) the path argument can be an RDD of strings: path : str or list string, or list of strings, for input path(s), or RDD of Strings storing CSV rows. With that, you may use spark.sparkContext.textFi...