Best Practices for Daily Source-to-Bronze Data Ingestion in Databricks

[JissMathew]

How can we effectively manage source-to-bronze data ingestion from a project perspective, particularly when considering daily scheduling strategies using either Auto Loader or Serverless Warehouse COPY INTO commands?

[BigRoux]

JissMathew, you have options when ingesting data from Cloud Storage into Delta Lake.

Your first approach should be with Delta Live Tables, it will be the easiest and most cost effective approach.  Among other benefits it will manage the infrastructure for you so that you don't have to think about what type of compute resources you should be using.

Another option is to create structure streaming jobs and schedule using Workflows.  Like DLT, you can leverage autoloader to pull files from cloud storage in a very efficient way.  

DLT is the best way forward though.

Cheers, Louis.

[JissMathew]

@BigRoux  thank you for your reply , for implementing dlt is that we need multi node cluster ?

[BigRoux]

No, it is not a strict requirement. You can have a single node job cluster run the job if the job is small.