Utilize Uber’s Petastorm https://github.com/uber/petastorm to read in and generate a parquet dataset from the image data. The petastorm API’s can then be used to generate a TF Dataset, etc.
Only when you have a gradient-descent problem. Pytorch and Tensorflow are the only candidate frameworks to use here. When using Horovod, start with single node, multi-GPU and measure training performance. If this is not sufficient, look at a multi-no...
