Currently using the Serverless compute but ideally my custom library shouldn't limit the cluster choice.

It feels like I only have some options:

- upload data to volume and run COPY INTO

- upload data to DBFS and run COPY INTO

- or leverage the pre-configured spark client session, and use spark in my custom library

I am not a databricks expert - please correct me if I am wrong.