Ok. I have created a small and simple example.... But I want to use DeepAR and Temporal Fusion Transformer model on 6.000.000 row x 155 columns dataset.

It is still not clear that when I want to create data chunks, e.g. I want to predict the next day's data based on the previous week's data (7 days), how do I do this with Spark?

With normal pilinle, it looks like I'm making the 2D data into 3D, using numpy, or e.g. tensorflow.keras.preprocessing.timeseries_dataset_from_array.