I am using a framework and i have a query where i am doing,
df = seg_df.select(*).write.option("compression", "gzip') and i am getting below error,
When i don't do the write.option i am not getting below error. Why is it giving me repartition error. What is that.
/tmp/tmpy1752o0x.py in runner(params, bridge, paths, date_graph, date)
97 date=date,
98 **kw)
---> 99 load.write(df,
100 table_name=table,
101 layer=current_layer,
/local_disk0/spark-c24724ad-99c7-4934-8aad-c9082a731e34/userFiles-e9f6ed1f-8660-40d0-be02-e4ab3d83f5e4/pipeline.zip/load.py in write(df, table_name, layer, date, spark, date_graph)
757 date_graph=date_graph)
758 else:
--> 759 single_write(df,
760 entry,
761 layer=layer,
/local_disk0/spark-c24724ad-99c7-4934-8aad-c9082a731e34/userFiles-e9f6ed1f-8660-40d0-be02-e4ab3d83f5e4/pipeline.zip/load.py in single_write(df, entry, layer, date, spark, tablename, date_graph)
615 data_name = final_entry_cp['dbtable']
616 if 'repartition' in final_entry_cp:
--> 617 df = df.repartition(final_entry_cp['repartition'])
618 if 'partitionBy' in entry:
619 df = df.repartition(final_entry_cp['partitionBy'])
AttributeError: 'NoneType' object has no attribute 'repartition'
