Databricks Community

imgaboy · 03-10-2022

I am still lost on the Spark and Deep Learning model.If I have a (2D) time series that I want to use for e.g. an LSTM model. Then I first convert it to a 3D array and then pass it to the model. This is normally done in memory with numpy. But what hap...

imgaboy · 03-08-2022

I have a large time series with many measuring stations recording the same 5 data (Temperature, Humidity, etc.) I want to predict a future moment with a time series model, for which I pass the data from all the measuring stations to the Deep Learning...

imgaboy · 03-10-2022

My problem is how to add/transform this dataset to the model from Spark data frame, without reading all data to the memory. The sample code above shows that problem...

imgaboy · 03-10-2022

A simple DL model like an image classifier uses more than 65.000-pixel point as an input...So it is not a mutch data for Neural Network....

imgaboy · 03-10-2022

After the pivot, I have a table with 30.000 rows and 16.000 columns. PySpark does not "love" that amount of columns......

imgaboy · 03-10-2022

Ok. I have created a small and simple example.... But I want to use DeepAR and Temporal Fusion Transformer model on 6.000.000 row x 155 columns dataset. It is still not clear that when I want to create data chunks, e.g. I want to predict the next da...

imgaboy · 03-10-2022

Thank you I have made a solution based on your idea.

Databricks Community

User Stats

User Activity

Spark with LSTM

pySpark Dataframe to DeepLearning model

Re: Spark with LSTM

Re: Spark with LSTM

Re: Spark with LSTM

Re: Spark with LSTM

Re: pySpark Dataframe to DeepLearning model