cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Solution for ConnectException error: This is often caused by an OOM error that causes the connection to the Python REPL to be closed. Check your query's memory usage.

Satty
New Contributor

When ever I am trying to run and load multiple files in single dataframe for processing (overall file size is more than 15 gb in single dataframe at the end of the loop, my code is crashing everytime with the below error...

ConnectException error: This is often caused by an OOM error that causes the connection to the Python REPL to be closed. Check your query's memory usage.

Please help me to fix it. Below is my code

df2= pd.DataFrame()

for i in range(0, k):

    df1= pd.DataFrame()

       

    for j in pd.date_range(start_date, periods=5):

      print(i, start_date)

      path = r'/dbfs/mnt/xxxx/***/Ixxxx/***/'

      path1 = os.path.join(path,'XXXX_'+ start_date +'.csv')

      if os.path.isfile(path1):

        df= pd.read_csv(path1, low_memory=False)

        df= df.drop(['Var1', 'Var2', 'Var3'], axis=1)

        df= df.drop_duplicates(keep='first')

        df.reset_index(drop=True, inplace=True)

        df.set_index('VmsNo', inplace=True)

        df1= df1.append(df)

      start_date = (pd.Timestamp(start_date)- pd.DateOffset(days=1)).strftime('%Y%m%d')

    df2= df2.append(df1)        

1 REPLY 1

pvignesh92
Honored Contributor

@Satish Agarwal​ It seems your system memory is not sufficient to load the 15GB file. I believe you are using Python Pandas data frame for loading 15GB file and not using Spark. Is there any particular reason that you cannot use Spark for this.

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group