Witold
Databricks Partner

Multiple things.

  1. First very obvious thing: "5000000" - It's not surprising that you run OOM when loading such a huge amount of records.
  2. The DBR has a built-in MySQL driver, use it instead
  3. Use fetchSize to control the number of records. Start e.g. with 1000 and measure the performance. Adapt it if needed
  4. If the source table has a partition column, use it as described here.