https://community.databricks.com/t5/machine-learning/how-do-i-efficiently-read-image-data-for-a-deep-learning-model/m-p/25858#M1421 <P>Our documentation provides nice examples of preparing image data for training and inference.</P><P><B>Training</B>: See docs for <A href="https://docs.databricks.com/applications/machine-learning/load-data/ddl-data.html#petastorm-recommended" alt="https://docs.databricks.com/applications/machine-learning/load-data/ddl-data.html#petastorm-recommended" target="_blank">AWS</A>, <A href="https://docs.microsoft.com/en-us/azure/databricks/applications/machine-learning/load-data/petastorm" alt="https://docs.microsoft.com/en-us/azure/databricks/applications/machine-learning/load-data/petastorm" target="_blank">Azure</A>, <A href="https://docs.gcp.databricks.com/applications/machine-learning/load-data/ddl-data.html#petastorm-recommended" alt="https://docs.gcp.databricks.com/applications/machine-learning/load-data/ddl-data.html#petastorm-recommended" target="_blank">GCP</A></P><P><B>Inference</B>: See reference solution for <A href="https://docs.databricks.com/applications/machine-learning/reference-solutions/images-etl-inference.html" alt="https://docs.databricks.com/applications/machine-learning/reference-solutions/images-etl-inference.html" target="_blank">AWS</A>, <A href="https://docs.microsoft.com/en-us/azure/databricks/applications/machine-learning/reference-solutions/images-etl-inference" alt="https://docs.microsoft.com/en-us/azure/databricks/applications/machine-learning/reference-solutions/images-etl-inference" target="_blank">Azure</A>, <A href="https://docs.gcp.databricks.com/applications/machine-learning/reference-solutions/images-etl-inference.html" alt="https://docs.gcp.databricks.com/applications/machine-learning/reference-solutions/images-etl-inference.html" target="_blank">GCP</A></P> Tue, 08 Jun 2021 19:46:46 GMT Joseph_B 2021-06-08T19:46:46Z https://community.databricks.com/t5/machine-learning/how-do-i-efficiently-read-image-data-for-a-deep-learning-model/m-p/25856#M1419 <P>How do I efficiently read image data for a deep learning model?</P> Mon, 07 Jun 2021 18:13:30 GMT https://community.databricks.com/t5/machine-learning/how-do-i-efficiently-read-image-data-for-a-deep-learning-model/m-p/25856#M1419 User16788317466 2021-06-07T18:13:30Z https://community.databricks.com/t5/machine-learning/how-do-i-efficiently-read-image-data-for-a-deep-learning-model/m-p/25857#M1420 <P>Utilize Uber’s Petastorm <A href="https://github.com/uber/petastorm" alt="https://github.com/uber/petastorm" target="_blank"><U></U></A><A href="https://github.com/uber/petastorm" target="test_blank">https://github.com/uber/petastorm</A> to read in and generate a parquet dataset from the image data. The petastorm API’s can then be used to generate a TF Dataset, etc.</P><P></P> Mon, 07 Jun 2021 18:13:53 GMT https://community.databricks.com/t5/machine-learning/how-do-i-efficiently-read-image-data-for-a-deep-learning-model/m-p/25857#M1420 User16788317466 2021-06-07T18:13:53Z https://community.databricks.com/t5/machine-learning/how-do-i-efficiently-read-image-data-for-a-deep-learning-model/m-p/25858#M1421 <P>Our documentation provides nice examples of preparing image data for training and inference.</P><P><B>Training</B>: See docs for <A href="https://docs.databricks.com/applications/machine-learning/load-data/ddl-data.html#petastorm-recommended" alt="https://docs.databricks.com/applications/machine-learning/load-data/ddl-data.html#petastorm-recommended" target="_blank">AWS</A>, <A href="https://docs.microsoft.com/en-us/azure/databricks/applications/machine-learning/load-data/petastorm" alt="https://docs.microsoft.com/en-us/azure/databricks/applications/machine-learning/load-data/petastorm" target="_blank">Azure</A>, <A href="https://docs.gcp.databricks.com/applications/machine-learning/load-data/ddl-data.html#petastorm-recommended" alt="https://docs.gcp.databricks.com/applications/machine-learning/load-data/ddl-data.html#petastorm-recommended" target="_blank">GCP</A></P><P><B>Inference</B>: See reference solution for <A href="https://docs.databricks.com/applications/machine-learning/reference-solutions/images-etl-inference.html" alt="https://docs.databricks.com/applications/machine-learning/reference-solutions/images-etl-inference.html" target="_blank">AWS</A>, <A href="https://docs.microsoft.com/en-us/azure/databricks/applications/machine-learning/reference-solutions/images-etl-inference" alt="https://docs.microsoft.com/en-us/azure/databricks/applications/machine-learning/reference-solutions/images-etl-inference" target="_blank">Azure</A>, <A href="https://docs.gcp.databricks.com/applications/machine-learning/reference-solutions/images-etl-inference.html" alt="https://docs.gcp.databricks.com/applications/machine-learning/reference-solutions/images-etl-inference.html" target="_blank">GCP</A></P> Tue, 08 Jun 2021 19:46:46 GMT https://community.databricks.com/t5/machine-learning/how-do-i-efficiently-read-image-data-for-a-deep-learning-model/m-p/25858#M1421 Joseph_B 2021-06-08T19:46:46Z