mmayorga
Databricks Employee
Databricks Employee

Hi @Michał ,

One detail/feature to consider when working with Declarative Pipelines is that they manage and auto-tune configuration aspects, including rate limiting (maxBytesPerTrigger or maxFilesPerTrigger). Perhaps that's why you could not see this configured behavior; outside Declarative Pipelines, it should be fine to do so.

Yes, Declarative Pipelines use checkpoints for each flow writing to streaming tables. Unfortunately, these are not accessible as the service handles them for usHere are more details about "Recover a Lakeflow Declarative Pipelines from streaming checkpoint failure.

Also, remember that with Declarative Pipelines, you get an event_log table, which you can configure in the advanced settings. This will get you further details about the specific flow between your tables and more (auditing, quality, etc). Hopefully, this can help you determine what happened with your pipeline on your first step/task. Perhaps the data is not following your schema, even with that minimal processing that you mentioned? 

We recommend following the medallion architecture, where the bronze table(raw data) will allow you to examine the data as it comes, then apply transformations into the silver layer and capture these possible issues.

I hope this helps!