Facing issue while accessing DLT pipeline event logs.

PawanB — Thu, 20 Feb 2025 06:23:03 GMT

while accessing event logs created by dlt pipeline, i am getting below error.

[INTERNAL_ERROR] The Spark SQL phase analysis failed with an internal error. You hit a bug in Spark or the Spark plugins you use. Please, report this bug to the corresponding communities or vendors, and provide the full stack trace. SQLSTATE: XX000

Re: Facing issue while accessing DLT pipeline event logs.

Brahmareddy — Mon, 24 Feb 2025 19:17:26 GMT

Hi Pawan,

How are you doing today? As per my understanding, This error likely indicates an internal issue with Databricks, Delta Live Tables (DLT), or a Spark plugin. First, check if there are any known issues in Databricks release notes or community forums. If it’s a transient issue, try restarting the cluster and rerunning the query. Ensure you're querying the correct event log table with SELECT * FROM event_log WHERE pipeline_id = '<your_pipeline_id>'. If the issue persists, there might be corrupt data, so try filtering older records or limiting the query. Also, upgrading to the latest Databricks LTS (Long-Term Support) runtime might help with compatibility issues. If nothing works, raising a Databricks support ticket with the full stack trace is the best option. Hope this helps!

Give a try and ley me know.

Regards,

Brahma

topic Re: Facing issue while accessing DLT pipeline event logs. in Data Engineering

Facing issue while accessing DLT pipeline event logs.

Re: Facing issue while accessing DLT pipeline event logs.