To run an SDP (Spark Declarative Pipeline) in parallel with dynamic parameters, you need to understand that SDP is "smart"—it builds a dependency graph and runs everything it can at the same time by default.

Here is a simple breakdown of how to handle your specific questions:

1. Running in Parallel
You don't actually need to write "parallel code." Because SDP is declarative, the engine looks at your @sdp.table definitions. If Table A and Table B don't depend on each other, SDP will automatically trigger them in parallel to save time.

2. Handling Dynamic Parameters
To pass parameters into your pipeline without them getting messy or overwritten, the best way is to use Spark Configurations.

The Overwrite Issue: You’re right—if you use the same name at the Job and Pipeline levels, the Job level usually wins.

3. Do you always need a Delta Live Table?
Nope! While SDP is the engine behind Delta Live Tables (DLT) in Databricks, the open-source version is flexible. You can write to Parquet, Iceberg, or even just Temporary Views if you don't want to save the data permanently.

However, using Delta is usually recommended because it supports "Time Travel" and "Z-Ordering," which makes your queries much faster later on.

You can refer to the official documents for more details.

https://docs.databricks.com/aws/en/ldp/parameters

https://spark.apache.org/docs/latest/declarative-pipelines-programming-guide.html

Thank you!