cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
cancel
Showing results for 
Search instead for 
Did you mean: 

Save data from Spark DataFrames to TFRecords

THIAM_HUATTAN
Contributor III

https://docs.microsoft.com/en-us/azure/databricks/_static/notebooks/deep-learning/tfrecords-save-loa...

I could not run the Cell # 2

java.lang.ClassNotFoundException: 
---------------------------------------------------------------------------
Py4JJavaError                             Traceback (most recent call last)
<command-4336535992800220> in <module>
      3 path = '/ml/flowersData/df-{}.tfrecord'.format(name_uuid)
      4 print(path)
----> 5 df.limit(100).write.format("tfrecords").mode("overwrite").save(path)
 
/databricks/spark/python/pyspark/sql/readwriter.py in save(self, path, format, mode, partitionBy, **options)
    738             self._jwrite.save()
    739         else:
--> 740             self._jwrite.save(path)
    741 
    742     @since(1.4)
 
/databricks/spark/python/lib/py4j-0.10.9.1-src.zip/py4j/java_gateway.py in __call__(self, *args)
   1302 
   1303         answer = self.gateway_client.send_command(command)
-> 1304         return_value = get_return_value(
   1305             answer, self.gateway_client, self.target_id, self.name)
   1306 

Can anyone help?

1 ACCEPTED SOLUTION

Accepted Solutions

jose_gonzalez
Moderator
Moderator

Hi @THIAM HUAT TAN​,

Which DBR version are you using? are you using the ML runtime?

View solution in original post

2 REPLIES 2

jose_gonzalez
Moderator
Moderator

Hi @THIAM HUAT TAN​,

Which DBR version are you using? are you using the ML runtime?

yes, the issue arise from the cluster which is a non-ML runtime. Thanks for your comments.

Welcome to Databricks Community: Lets learn, network and celebrate together

Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections. 

Click here to register and join today! 

Engage in exciting technical discussions, join a group with your peers and meet our Featured Members.