dbt writing parquet from Volumes to Catalog schema

ruoyuqian — Tue, 30 Jul 2024 01:55:59 GMT

I have ran into a weird situation, so I uploaded few parquet files (about 10) for my sales data into the Volume in my catalog, and run dbt againt it , dbt went successful and table was able to be created however when i upload a lot more parquet files it just fails, and says

[TASK_WRITE_FAILED] Task failed while writing rows to abfss://datalake-raw-dev@xxxx.dfs.core.windows.net/__unitystorage/schemas/xxxx/tables/zzzz. SQLSTATE: 58030

why is it that for small amount of parquet it can be processed while a large amount of parquet(about 2500) it does not work.

Re: dbt writing parquet from Volumes to Catalog schema

NandiniN — Thu, 09 Jan 2025 11:23:53 GMT

When dealing with a large number of Parquet files (about 2500 in your case), the system might be running into resource limitations or timeouts. This can happen due to the sheer volume of data being processed at once.

The failure might be due to insufficient resources allocated to handle the large volume of data. Ensure that your cluster has enough resources (CPU, memory) to handle the load.

topic dbt writing parquet from Volumes to Catalog schema in Data Engineering

dbt writing parquet from Volumes to Catalog schema

Re: dbt writing parquet from Volumes to Catalog schema