Databricks Community

Hi Team, I am working with huge volume of data (50GB) and i decompose the time series data using the statsmodel.Having said that the major challenge i am facing is the compatibility of the pyspark dataframe with the machine learning algorithms. altho...

Hello everyone ,Here are the sequence of steps i have followed:1. I have used petastorm to convert the spark dataframe to tf.datasetimport numpy as np# Read the Petastorm dataset and convert it to TensorFlow Datasetwith converter.make_tf_dataset() as...