topic Re: COPY INTO size limit in Data Engineering

COPY INTO size limit

DBUser2 — Mon, 11 Nov 2024 19:21:57 GMT

I'm using the COPY INTO command to ingest data into a delta table in my Azure Databricks instance. Sometime I get a timeout error running this command. Is there a limit on the size of the data that can be ingested using "COPY INTO" or limit on the number of files that can be ingested at a time?

Thanks

Re: COPY INTO size limit

Alberto_Umana — Mon, 11 Nov 2024 20:16:21 GMT

Hello @DBUser2,

The COPY INTO command does not have a specific documented limit on the size of the data or the number of files that can be ingested at a time. Timeout errors can occur due to network issues, resource limitations, or long-running operations. Are you running the commands on a warehouse or over notebook?

Looking at cluster/warehouse metrics would be a good way to start investigating.

Also, do you have the statementID is executed via warehouse?

Re: COPY INTO size limit

DBUser2 — Mon, 11 Nov 2024 22:12:29 GMT

Thanks for replying, i'm running the command on sql warehouse.