It turns out this problem was caused by a package that was pip installed using an init script. This package had for some reason started pulling in pandas 3.x (despite the fact that the package itself had not been updated), and our Databricks contact informed us that DLT does not support Pandas 3 at this time (at least until the next version of the DBR). Once we pinned the pandas version to < 3 then the pipelines started working again.

No idea why pandas 3 was suddenly chosen, nor why the error message is so cryptic, but glad we found a solution in the end!