JRL
New Contributor III

On a sql server, there are wait states. Wait states occur when several processors (vCPUs) are processing and several threads are working through the processors. A longer running thread that has dependencies, can cause the thread that may have begun on a different process, but is dependent on the one that has the longer running thread to complete, will cause a wait state and slow down the dependent query. From the sysadmin POV, this could also be seen as hard faults in memory. The issue above from a SQL perspective is to increase the maximum bound which add endpoints (threads) and logically create additional flows - more streams, faster flow. From the sysadmin POV, adding additional clusters to increase the size of the endpoint would also solve the problem because the amount of memory allows for swifter execution. In either case, it should resolve the issue. It would make more sense to allow Azure or AWS to sort this out with a Serverless feature that is Reliably Optimizied. However, that feature will simply perform the feat of scaling the cluster. Assuming AWS/Azure have a handle on it, and their methodology is to scale the cluster, the better answer..., based on their handling of the situation, is to scale upward with the cluster. I suspect the answer here is the Databricks people aren't always thinking like SQL Server database administrators. They have lots of cluster power. Frankly, if it were my issue, I would try the maximum bound increase first. If that didn't work then I'd go for the increased cluster size. But my thinking is a cost optimization. There is not a part of the question that asks how to do this in a cost effective way. So - final answer - C: boost the cluster.