- 3731 Views
- 7 replies
- 0 kudos
data from 10 BLOB containers and multiple hierarchical folders(every day and every hour folders) in each container to Delta lake table in parquet format - Incremental loading for latest data only insert no updates
I am able to load data for single container by hard coding, but not able to load from multiple containers. I used for loop, but data frame is loading only last container's last folder record only.Here one more issue is I have to flatten data, when I ...
- 3731 Views
- 7 replies
- 0 kudos
Latest Reply
- 0 kudos
for sure function (def) should be declared outside loop, move it after importing libraries,logic is a bit complicated you need to debug it using display(Flatten_df2) (or .show()) and validating json after each iteration (using break or sleep etc.)
- 0 kudos
- 920 Views
- 0 replies
- 0 kudos
- 920 Views
- 0 replies
- 0 kudos