1. Incrementally load data from Table A as a batch: You can use Delta Live Tables' built-in capabilities for reading data from Delta tables, including support for incremental loading. -> Which build-in capabilities are you talking about? From my understanding of the documentation, the only way you can load the table incrementally is by using readStream (you said it so yourself: readStream method in Delta Live Tables), and that provides a streaming DataFrame, not batch. As foreachbatch cannot be called in delta live tables, the only way is to use @apply_change_data and accumulate it in another table and then use it from there. Is there another way? Can you provide an example of how to load incremental data as a batch? You also mentioned configure the batch mode settings such as the maximum number of rows per batch, the maximum duration per batch, and the mode of data ordering, but this only configures settings for micro-batches in streaming, it does not allow us to handle the streaming pipeline at once or do transformations per micro batch using batch spark operations.
2. Handling previously failed batches: Delta Live Tables allows you to specify the start position for reading data from a Delta table using the startingVersion option. <- For this you need to build your own check-pointing logic. This also means that you create a circular dependency as at the end of the pipeline you have to write somewhere the version of the last successful run, then load it at startup. You also have to somehow involve the TRUNCATE operation if you are using @apply_change_data so that we can reset incremental loads if pipeline was successful. This logic is very cumbersome to implement (I am currently testing this as a solution to what I want to do, but this seems more like a hack than a proper solution. If for any reason you want to run the pipelines on a different start/end versions of the input tables you will lose the accumulated data. There might be workarounds, but I get the feeling that I am just hacking around limitations in delta live tables)
3. Yea. This we know. Problem is that if in the previous pipeline you receive a streaming DataFrame, you are limited in what aggregation you can do, either by memory limitation if historical computation is required (watermark will be big) or just by the fact that any aggregation in streaming requires a partitioning by time, limiting what you can do.
4. You are very limited in what output modes you can actually use as in delta live tables you shouldn't call write or writeStream directly. We do have a lot of upserts into our pipeline logic so that we only care about rows and aggregations which change. As @apply_change_data does not allow custom merge (or even at least just detecting that a row has not changed), you have to join with the existing data yourself and basically reimplement the upsert functionality yourself. This amounts to you basically rebuilding the wheel that Databricks already built.
5. Do checkpoints in streams get rolled back if a task has failed? The recovery mechanism mentioned is just a retry mechanism, not a recovery one. Recovery mechanism should support rollback, not only retry. From my understanding of the documentation, there is no rollback mechanism in the pipelines in case of failure, but I can be wrong about this. Is there a part in the documentation that says anithing about rolling back checkpoints and data?

By following these best practices and utilizing the features of Delta Live Tables, you can implement a robust pipeline that handles new data as a Spark batch, recovers from failures, and provides reliable data processing for your use case. -> We would love to but it seems we are very limited in what we can do with best practices alone, especially if the documentation lacks proper examples for said best practices.