- 4813 Views
- 6 replies
- 3 kudos
Resolved! Changing shuffle.partitions with spark.conf in a spark stream - isn't respected even after a checkpoint
Question about spark checkpoints and offsets in a running streamWhen the stream started I needed tons of partitions, so we've set it with spark.conf to 5000As expected offsets in the checkpoint contain this info and the job used this value. Then we'...
- 4813 Views
- 6 replies
- 3 kudos
- 3 kudos
@Jose Gonzalez​ thanks for that information! This is super useful. I was struggling why my streaming still using 200 partitions. This is quite a paint for me because changing checkpoint will insert all data from the source. Do you know where this can...
- 3 kudos
- 1654 Views
- 0 replies
- 0 kudos
How can I see details of a previous version of a table (e.g. number of files)?
I know that I can do a DESCRIBE DETAIL on a table to get current delta table version details. If I want to get these same details on a previous version, how can I do that?
- 1654 Views
- 0 replies
- 0 kudos