Sometimes for such a simple data transformation like merging I use indeed Data Factory but as it is written on top of Spark it will be similar.

More workers and parallel read jobs will help. For example in data factory you can use "Run in parallel" and "Set partitioning" options.

In databricks it will be faster with autoload (you can always use stream trigger once to run as batch) https://docs.microsoft.com/en-us/azure/databricks/spark/latest/structured-streaming/auto-loader-gen2

Setting schema instead of inferSchema will help to avoid double scan.

Always partitioning is a key so files will be loaded from different directories in parallel: