Hey @ShlomoSQM, looks like @shan_chandra suggested a feasible solution, just to add a little more context this is how you can achieve the same if you have a column that can help you identify what is type1 and type 2
file_type1_stream = readStream.option("cloudFiles.format", "parquet") \
.load("/mnt/source_volume") \
.filter(col("file_type") == "type1") # Assuming a column indicating file type
file_type2_stream = readStream.option("cloudFiles.format", "parquet") \
.load("/mnt/source_volume") \
.filter(col("file_type") == "type2")
file_type1_stream.writeStream.format("delta") \
.option("checkpointLocation", "/mnt/checkpoints/type1") \
.toTable("catalog.volume.table_name1")
file_type2_stream.writeStream.format("delta") \
.option("checkpointLocation", "/mnt/checkpoints/type2") \
.toTable("catalog.volume.table_name2")
Leave a like if this helps! Kudos,
Palash