Hi,

We use lakehouse federation to connect to a database.
A performance recommendation is to use 'fetchSize':
Lakehouse Federation performance recommendations - Azure Databricks | Microsoft Learn

SELECT * 
FROM mySqlCatalog.schema.table 
WITH ('fetchSize' 100000)

But as far as I can find it's only possible in SQL.

As we would like to do some further column transformations on it, we'd like to use pyspark.

I know you can do spark.sql("....") as well, but is there any way to specify the fetch size in pure pyspark, instead of SQL?