cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
cancel
Showing results for 
Search instead for 
Did you mean: 

AttributeError: 'NoneType' object has no attribute 'repartition'

amitdatabricksc
New Contributor II

I am using a framework and i have a query where i am doing,

df = seg_df.select(*).write.option("compression", "gzip') and i am getting below error,

When i don't do the write.option i am not getting below error. Why is it giving me repartition error. What is that.

/tmp/tmpy1752o0x.py in runner(params, bridge, paths, date_graph, date)

   97             date=date,

   98             **kw)

---> 99       load.write(df,

  100             table_name=table,

  101             layer=current_layer,

/local_disk0/spark-c24724ad-99c7-4934-8aad-c9082a731e34/userFiles-e9f6ed1f-8660-40d0-be02-e4ab3d83f5e4/pipeline.zip/load.py in write(df, table_name, layer, date, spark, date_graph)

  757             date_graph=date_graph)

  758   else:

--> 759     single_write(df,

  760           entry,

  761           layer=layer,

/local_disk0/spark-c24724ad-99c7-4934-8aad-c9082a731e34/userFiles-e9f6ed1f-8660-40d0-be02-e4ab3d83f5e4/pipeline.zip/load.py in single_write(df, entry, layer, date, spark, tablename, date_graph)

  615     data_name = final_entry_cp['dbtable']

  616   if 'repartition' in final_entry_cp:

--> 617     df = df.repartition(final_entry_cp['repartition'])

  618   if 'partitionBy' in entry:

  619     df = df.repartition(final_entry_cp['partitionBy'])

AttributeError: 'NoneType' object has no attribute 'repartition'

2 REPLIES 2

Hubert-Dudek
Esteemed Contributor III

When you write dataframe please don't assign it to new name ( ommit df = )

jose_gonzalez
Moderator
Moderator

hi @AMIT GADHAVI​ ,

could you provide more details? for example, what is your data source? how do you repartition? etc