We had a similar experience where the initialisation and setting up tables step took as long as 2 hours before the pipeline starts. 

It gotten worse when we migrated the pipeline to Unity Catalog. When we requested support, one of the first recommendations was to remove all "directory listing" operations. 

First steps of our pipeline were traversing directories and reading CSVs into dataframes. Converting that into streaming steps brought the total initialisation + setting up tables step down to <10 minutes. 

However it is a big mystery why directory listing performs terribly in Delta Live Tables, I could never get an answer to it.