AttributeError: 'NoneType' object has no attribute 'repartition'

amitdatabricksc — Sat, 16 Oct 2021 18:53:55 GMT

I am using a framework and i have a query where i am doing,

df = seg_df.select(*).write.option("compression", "gzip') and i am getting below error,

When i don't do the write.option i am not getting below error. Why is it giving me repartition error. What is that.

/tmp/tmpy1752o0x.py in runner(params, bridge, paths, date_graph, date)

97 date=date,

98 **kw)

---> 99 load.write(df,

100 table_name=table,

101 layer=current_layer,

/local_disk0/spark-c24724ad-99c7-4934-8aad-c9082a731e34/userFiles-e9f6ed1f-8660-40d0-be02-e4ab3d83f5e4/pipeline.zip/load.py in write(df, table_name, layer, date, spark, date_graph)

757 date_graph=date_graph)

758 else:

--> 759 single_write(df,

760 entry,

761 layer=layer,

/local_disk0/spark-c24724ad-99c7-4934-8aad-c9082a731e34/userFiles-e9f6ed1f-8660-40d0-be02-e4ab3d83f5e4/pipeline.zip/load.py in single_write(df, entry, layer, date, spark, tablename, date_graph)

615 data_name = final_entry_cp['dbtable']

616 if 'repartition' in final_entry_cp:

--> 617 df = df.repartition(final_entry_cp['repartition'])

618 if 'partitionBy' in entry:

619 df = df.repartition(final_entry_cp['partitionBy'])

AttributeError: 'NoneType' object has no attribute 'repartition'

Re: AttributeError: 'NoneType' object has no attribute 'repartition'

Hubert-Dudek — Sun, 17 Oct 2021 12:14:48 GMT

When you write dataframe please don't assign it to new name ( ommit df = )

Re: AttributeError: 'NoneType' object has no attribute 'repartition'

jose_gonzalez — Mon, 18 Oct 2021 22:21:28 GMT

hi @AMIT GADHAVI ,

could you provide more details? for example, what is your data source? how do you repartition? etc

topic AttributeError: 'NoneType' object has no attribute 'repartition' in Data Engineering

AttributeError: 'NoneType' object has no attribute 'repartition'

Re: AttributeError: 'NoneType' object has no attribute 'repartition'

Re: AttributeError: 'NoneType' object has no attribute 'repartition'