@Pedro Barbosa​ :

It seems like you are running out of memory when trying to convert the PySpark dataframe to an H2O frame. One possible approach to solve this issue is to partition the PySpark dataframe before converting it to an H2O frame.

You can use the repartition() method of PySpark dataframes to split your dataframe into smaller partitions and then convert each partition into an H2O frame separately. You can then concatenate the resulting H2O frames to create the final H2O frame.

Here is an example code snippet that shows how to partition the PySpark dataframe and convert each partition to an H2O frame:

# Imports
 
import h2o
from pysparkling import *
from pyspark.sql import SparkSession
 
# Creates an H2OContext Object.
hc = H2OContext.getOrCreate(spark)
 
# Read the Data from Delta Table to PySpark Dataframe
df = spark.read.format("delta").load("path/to/delta_table")
 
# Partition the dataframe
num_partitions = 10
df = df.repartition(num_partitions)
 
# Convert each partition to an H2O frame
h2o_frames = []
for i in range(num_partitions):
    partition_df = df.where(df.partition_id() == i)
    h2o_frame = hc.asH2OFrame(partition_df)
    h2o_frames.append(h2o_frame)
 
# Concatenate the H2O frames to create the final H2O frame
h2o_df = h2o.frames.concat(h2o_frames)
 
# H2O Model for Anomaly Detection
isolation_model = H2OIsolationForestEstimator(model_id, ntrees, seed)
isolation_model.train(training_frame = h2o_df)
 
# Making predictions
predictions = isolation_model.predict(h2o_df)

This approach should help you avoid running out of memory when converting large PySpark dataframes to H2O frames.