There are several ways to do this, but I personally like to use the sql statement API (see docs here : https://docs.databricks.com/sql/admin/sql-execution-tutorial.html)

Coupled with a serverless sql warehouse, this allows you to run fast queries on your data (for example a simple select *) and retrieve the data for you to process however you want.