When I read managed table in #databricks# i can see big differences in time spent. Small test table with just 2 records is once loaded in 3 seconds and another time in 30 seconds. Reading table_change for this tinny table took 15 minutes. Don't know if this has any explanation or cluster setting is somehow wrong.
- storage: ADLS gen2
- cluster config:
- access mode: shared
- runtime: 13.2 (includes Apache Spark 3.4.0, Scala 2.12)
- worker: Standard_DS_v2 14GB, 4 Cores min 2 max 8
- driver: Standard_DS_v2
- enable autoscaling = true
- No other advanced options
Thanks.
