So if I'm understanding correctly, I can have a pipeline with a single step in it - SCD type 1 and it should work?

At the moment, it seems that if I clear the pipeline files in the abfs directory and start pipeline from scratch, it is able to carry out all of the steps if I run it as batch then stop. However the subsequent run fails. If I clear pipeline directory and run it again, it works.

So I'm suspecting that there is some trickery going on with keeping state / checkpointing that gets in the way.

Our pipelines are elaborate and use many steps. If I have to break out these pipelines into multiple orchestrated pieces, it will become very cumbersome.

But it would be nice to zero in on exactly what the issue is because we might be waiting for a fix that's not relevant. We will follow your advice anyway, just for evaluation purposes, but ultimately the pipeline would have to work regardless of what chain of operations is desired