Save data from Spark DataFrames to TFRecords

THIAM_HUATTAN — Wed, 29 Jun 2022 12:42:51 GMT

https://docs.microsoft.com/en-us/azure/databricks/_static/notebooks/deep-learning/tfrecords-save-load.html

I could not run the Cell # 2

java.lang.ClassNotFoundException: 
---------------------------------------------------------------------------
Py4JJavaError                             Traceback (most recent call last)
<command-4336535992800220> in <module>
      3 path = '/ml/flowersData/df-{}.tfrecord'.format(name_uuid)
      4 print(path)
----> 5 df.limit(100).write.format("tfrecords").mode("overwrite").save(path)
 
/databricks/spark/python/pyspark/sql/readwriter.py in save(self, path, format, mode, partitionBy, **options)
    738             self._jwrite.save()
    739         else:
--> 740             self._jwrite.save(path)
    741 
    742     @since(1.4)
 
/databricks/spark/python/lib/py4j-0.10.9.1-src.zip/py4j/java_gateway.py in __call__(self, *args)
   1302 
   1303         answer = self.gateway_client.send_command(command)
-> 1304         return_value = get_return_value(
   1305             answer, self.gateway_client, self.target_id, self.name)
   1306

Can anyone help?

Re: Save data from Spark DataFrames to TFRecords

jose_gonzalez — Tue, 05 Jul 2022 17:47:39 GMT

Hi @THIAM HUAT TAN,

Which DBR version are you using? are you using the ML runtime?

Re: Save data from Spark DataFrames to TFRecords

THIAM_HUATTAN — Wed, 06 Jul 2022 05:20:33 GMT

yes, the issue arise from the cluster which is a non-ML runtime. Thanks for your comments.

topic Re: Save data from Spark DataFrames to TFRecords in Data Engineering

Save data from Spark DataFrames to TFRecords

Re: Save data from Spark DataFrames to TFRecords

Re: Save data from Spark DataFrames to TFRecords