topic [TASK_WRITE_FAILED] Task failed while writing rows to abfss in Data Engineering

[TASK_WRITE_FAILED] Task failed while writing rows to abfss

FrancisApel — Wed, 13 Sep 2023 13:29:14 GMT

I am trying to insert into an already created delta table in Unity Catalog. I am getting the error:

[TASK_WRITE_FAILED] Task failed while writing rows to abfss://xxxx@xxxxxxxxxxxxxxxx.dfs.core.windows.net/__unitystorage/catalogs/xxxxxxxx-c6c8-45d8-ac3f-7ade50a0bbed/tables/76b5d99e-d355-4808-95d6-xxxxxxxxxxx.

The insert involves joining 17 tables (actually temporary view of tables) to get the data needed to insert. The code is created like follows:

INSERT INTO catalog1.schema1.product

(column1, column2, column3...)

SELECT a.col1, b.col2, c.col1...

FROM table1 a

LEFT JOIN table2 b on a.col1=b.col1

LEFT JOIN table3 C on a.col1=c.col1

...

I tried this with a warehouse cluster and a spark cluster and got the same error.

I also created a simple test table in the same UC catalog/schema and was able to insert into it without any issues (so it's not a security related issue).

Any help in how to further troubleshoot this issue would be greatly appreciated.

Re: [TASK_WRITE_FAILED] Task failed while writing rows to abfss

Atanu — Wed, 13 Sep 2023 15:28:04 GMT

Can you please share the driver log associated with this error. Would like to see the full error thread.

Re: [TASK_WRITE_FAILED] Task failed while writing rows to abfss

FrancisApel — Wed, 13 Sep 2023 20:01:01 GMT

Where would I find the driver log for Serverless SQL Warehouse?

Re: [TASK_WRITE_FAILED] Task failed while writing rows to abfss

chemajar — Thu, 14 Mar 2024 15:29:07 GMT

Hi, Francis.

Did you resolve the issue ?

Re: [TASK_WRITE_FAILED] Task failed while writing rows to abfss

NikunjKakadiya — Wed, 04 Sep 2024 02:46:50 GMT

Any chance this issue got resolved?
I am also seeing the same error when I am trying to incrementally read the system tables using the read stream method and writing it using the writestream method. This generally comes for the audit table but other tables runs fine without any issues.