topic Re: Model Training Data Adapter Error. in Data Engineering

Model Training Data Adapter Error.

NathanLaw — Thu, 19 May 2022 17:52:36 GMT

We are converting Pyspark dataframe to Tensorflow using PetaStorm and have encountered a “data adapter” error. What do you recommend for diagnosing and fixing this error?

https://docs.microsoft.com/en-us/azure/databricks/applications/machine-learning/load-data/petastorm

https://docs.microsoft.com/en-us/azure/databricks/_static/notebooks/deep-learning/petastorm-spark-converter-tensorflow.html

Thanks for help

Re: Model Training Data Adapter Error.

Anonymous — Mon, 06 Jun 2022 13:00:03 GMT

Hi @Nathan Law following up did you get a chance to check @Kaniz Fatma 's previous comments ?

Re: Model Training Data Adapter Error.

NathanLaw — Mon, 06 Jun 2022 13:30:49 GMT

Hi,

From the Petastorm example:

# Make sure the number of partitions is at least the number of workers which is required for distributed training.

I am testing an recommendation to not use Autoscaling. I'll report back with findings.

Nathan

Re: Model Training Data Adapter Error.

Anonymous — Tue, 19 Jul 2022 15:05:49 GMT

Hey there @Nathan Law

Hope all is well!

Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? It would be really helpful for the other members too.

We'd love to hear from you.

Cheers!

Re: Model Training Data Adapter Error.

NathanLaw — Tue, 19 Jul 2022 15:24:24 GMT

Making progress but still working through issues. I'll post findings when completed.

Re: Model Training Data Adapter Error.

Anonymous — Tue, 19 Jul 2022 15:35:54 GMT

Hey @Nathan Law

Thank you so much for getting back to us. We will await your response.

We really appreciate your time.