dbt writing parquet from Volumes to Catalog schema

[ruoyuqian]

I have ran into a weird situation, so I uploaded few parquet files (about 10) for my sales data into the Volume in my catalog, and run dbt againt it , dbt went successful and table was able to be created however when i upload a lot more parquet files it just fails, and says 

[TASK_WRITE_FAILED] Task failed while writing rows to abfss://datalake-raw-dev@xxxx.dfs.core.windows.net/__unitystorage/schemas/xxxx/tables/zzzz. SQLSTATE: 58030

why is it that for small amount of parquet it can be processed while a large amount of parquet(about 2500) it does not work.

[Kaniz_Fatma]

Hi @ruoyuqian, 

1. You can try increasing the cluster size or optimizing your Spark configurations to handle larger workloads.
2. Consider merging smaller files into larger ones or adjusting the partition size to optimize performance.
3. If your data is unevenly distributed, some partitions might be much larger than others, causing certain tasks to fail. You can try repartitioning your data to ensure a more even distribution.
4. Check your Spark and Databricks configurations. For example, increasing the spark.sql.shuffle.partitions setting can help with large datasets.

Would you like to dive deeper into any of these potential solutions?