Hubert-Dudek
Databricks MVP

Just use natively spark.sql this way code is much faster and simpler you use the power of RDD. All you need is

spark.sql("SELECT * FROM default.x LIMIT 2")


My blog: https://databrickster.medium.com/