Best Practices for Daily Source-to-Bronze Data Ingestion in Databricks
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-03-2024 10:46 PM
How can we effectively manage source-to-bronze data ingestion from a project perspective, particularly when considering daily scheduling strategies using either Auto Loader or Serverless Warehouse COPY INTO commands?
India .
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-04-2024 04:21 AM
JissMathew, you have options when ingesting data from Cloud Storage into Delta Lake.
Your first approach should be with Delta Live Tables, it will be the easiest and most cost effective approach. Among other benefits it will manage the infrastructure for you so that you don't have to think about what type of compute resources you should be using.
Another option is to create structure streaming jobs and schedule using Workflows. Like DLT, you can leverage autoloader to pull files from cloud storage in a very efficient way.
DLT is the best way forward though.
Cheers, Louis.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-04-2024 08:57 AM
@Louis_Frolio thank you for your reply , for implementing dlt is that we need multi node cluster ?
India .
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-04-2024 10:06 AM
No, it is not a strict requirement. You can have a single node job cluster run the job if the job is small.
