Delta Live Tables: control microbatch size
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
08-06-2024 10:53 PM
A delta live table pipeline reads a delta table on databricks. Is it possible to limit the size of microbatch during data transformation?
I am thinking about a solution used by spark structured streaming that enables control of batch size using:
.option("maxBytesPerTrigger", 104857600)
.option("maxFilesPerTrigger", 100) Is any similar option applicable?
Labels:
- Labels:
-
Workflows