https://community.databricks.com/t5/data-engineering/big-data-ingest-into-delta-lake/m-p/15238#M9594 <P>I have a feature table in BQ that I want to ingest into Delta Lake. This feature table in BQ has 100TB of data. This table can be partitioned by DATE.</P><P>What best practices and approaches can I take to ingest this 100TB? In particular, what can I do to distribute the write to Delta Lake to the workers and minimize memory pressure on the driver?</P> Thu, 30 Jun 2022 15:29:42 GMT spartakos 2022-06-30T15:29:42Z https://community.databricks.com/t5/data-engineering/big-data-ingest-into-delta-lake/m-p/15238#M9594 <P>I have a feature table in BQ that I want to ingest into Delta Lake. This feature table in BQ has 100TB of data. This table can be partitioned by DATE.</P><P>What best practices and approaches can I take to ingest this 100TB? In particular, what can I do to distribute the write to Delta Lake to the workers and minimize memory pressure on the driver?</P> Thu, 30 Jun 2022 15:29:42 GMT https://community.databricks.com/t5/data-engineering/big-data-ingest-into-delta-lake/m-p/15238#M9594 spartakos 2022-06-30T15:29:42Z