OK I see the problem. The issue is not databricks not being able to show these records.

The show command will run on the driver and for a lot of data this will give errors.

But there is a huge difference between showing data on screen and processing/writing them.

There is a reason there is a limit on the amount of records shown, as this is pretty expensive (cannot run in parallel too).

the display() command will default to 1000 records ,which can be overridden to 100K (or even a million, can't recall).