cancel
Showing results for 
Search instead for 
Did you mean: 
Warehousing & Analytics
Engage in discussions on data warehousing, analytics, and BI solutions within the Databricks Community. Share insights, tips, and best practices for leveraging data for informed decision-making.
cancel
Showing results for 
Search instead for 
Did you mean: 

Where are driver logs for SQL Pro Warehouse?

BamBam
New Contributor II

In an All-Purpose Cluster, it is pretty easy to get at the Driver logs.  Where do I find the Driver Logs for a SQL Pro Warehouse?  The reason I ask is because sometimes in a SQL Editor we get generic error messages like "Task failed while writing rows to abfss://...".  I am guessing the rest of the error message (i.e. full stack error) is in the Spark driver log.  Where are those driver logs located in SQL Pro Warehouse?

I was just in a Databricks Open Office Hours webinar and Clayton mentioned you can find this in the Query Profile.  I looked at the query profile and the 3 dots at the top-right corner only have the option to "Download" which downloads the query profile as JSON.

1 REPLY 1

BamBam
New Contributor II

Hi Kaniz,

Thanks for providing this information.  Yes, I am able to see the error in the Query Profile but it is the same error I saw in the SQL Editor UI.  Most errors in the SQL Editor are helpful, there are some that are vary vague and don't provide any insight without more log information.  For example:


"org.apache.spark.sql.execution.datasources.v2.DataSourceV2Relation cannot be cast to org.apache.spark.sql.execution.datasources.LogicalRelation"

 Or

[TASK_WRITE_FAILED] Task failed while writing rows to abfss://silver@<REDACTED>.dfs.core.windows.net/__unitystorage/catalogs/<REDACTED>/tables/<REDACTED>. 

I was able to write to the ADLS storage account just fine, it turned out to be a different error in the SQL.  So these generic errors didn't provide much insight and was looking for more detailed information in the SQL Pro Warehouse driver logs which sounds like is only available with the help of Databricks support.

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group