Big time differences in reading tables

alesventus
Contributor

When I read managed table in #databricks# i can see big differences in time spent. Small test table with just 2 records is once loaded in 3 seconds and another time in 30 seconds. Reading table_change for this tinny table took 15 minutes. Don't know if this has any explanation or cluster setting is somehow wrong.

  1. storage: ADLS gen2
  2. cluster config:
    • access mode: shared
    • runtime: 13.2 (includes Apache Spark 3.4.0, Scala 2.12)
    • worker: Standard_DS_v2 14GB, 4 Cores min 2 max 8
    • driver: Standard_DS_v2
  3. enable autoscaling = true
  4. No other advanced options

Thanks.