cancel
Showing results for 
Search instead for 
Did you mean: 
Community Platform Discussions
Connect with fellow community members to discuss general topics related to the Databricks platform, industry trends, and best practices. Share experiences, ask questions, and foster collaboration within the community.
cancel
Showing results for 
Search instead for 
Did you mean: 

Big time differences in reading tables

alesventus
Contributor

When I read managed table in #databricks# i can see big differences in time spent. Small test table with just 2 records is once loaded in 3 seconds and another time in 30 seconds. Reading table_change for this tinny table took 15 minutes. Don't know if this has any explanation or cluster setting is somehow wrong.

  1. storage: ADLS gen2
  2. cluster config:
    • access mode: shared
    • runtime: 13.2 (includes Apache Spark 3.4.0, Scala 2.12)
    • worker: Standard_DS_v2 14GB, 4 Cores min 2 max 8
    • driver: Standard_DS_v2
  3. enable autoscaling = true
  4. No other advanced options

Thanks.

 

 

0 REPLIES 0

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group