My source can only deliver CSV format (pipe delimited).

My source has the ability to generate multiple CSV files and transfer them to a single upload folder.

All rows must go to the same target bronze delta table.

I do not care about the order in which the rows are loaded.

The bronze target table columns can all be strings.

I am trying to find out if:

A: Is uploading multiple csv files and loading them with a single stream reader / stream writer statement the quickest way to load this data? In other words are multiple input files the way to introduce parallelism into the process? And if it is a single csv file, is it single threaded, is this correct?

B: Is there some optimal number of files and/or file sizes that the source data should be broken down into in order to achieve the ingestion speed. For example is 1-the number of worker nodes, or a multiple thereof, be the number of files I wish load? And does size matter, or is it just the number of input files matching a multiple of worker nodes all that counts?

C: Is there anything else I should be doing to improve the load times.