Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
Am trying to run a spark jdbc query and the read is running on only 1 worker and is taking a lot of time. Usually, it takes 4 minutes in aws lambda but now it is taking approx 10 minutes while using databricks. There are only 4m records (write is not taking long but read is consuming the max time)
