sajith_appukutt
Databricks Employee
Databricks Employee

There is no offset support yet. Here are a few possible workarounds

  • If you data is all in one partition ( rarely the case 🙂 ) , you could create a column with monotonically_increasing_id and apply filter conditions. if there are multiple partitions, monotonically_increasing_id won't be consecutive

  • Use except ( in your case sql equivalent of code below) . This however would be an expensive operation
df1 = df.limit(10); 
df2 = df.except(df1); 
df2.limit(20);