mathan_pillai
Databricks Employee
Databricks Employee

Hi @THIAM HUAT TAN

I don't think there is a way to specify that when reading it. However, after reading it, you can create monotonically increasing id (new column), and then filter for those ids that are greater than 4.

Alternatively you can apply take(4) and create rdd out of it. Then apply subtract transformation between the original rdd and the small rdd.

please let us know whether it works for you

Thanks