All,
I have the following code.
df_Warehouse_Utilization = (
spark.table("hive_metastore.dev_ork.bin_item_detail")
.join(df_DIM_Bins,col('bin_tag')==df_DIM_Bins.BinKey,'right')
.groupby(col('BinKey'))
.agg(count_distinct(when(col('serial_lot_tag')>0,col('serial_lot_tag'))).alias('SerialLotCount'),
count_distinct(when(col('tracking_tag')>0,col('tracking_tag'))).alias('Track Count'),
min(col('activation_time')).alias('OldestStorageTime')
)
.withColumn('BinUsedFlag',when(col('PartNumbers')>0,1).otherwise(0))
)
df_Warehouse_Utilization.write.mode("overwrite").format("delta").saveAsTable("yorkgold.df_DIM_Binsst")
display(df_Warehouse_Utilization)
when I write to table after the above transformation, the following error occurs.
AnalysisException: Found invalid character(s) among ' ,;{}()\n\t=' in the column names of your schema.
The issue is because I have the space in the column name (Track Count, please see the above highlighted), As soon as I rename column as (TrackCount), I am able to write to Hive.
So my question is, space is not allowed? Or is there any other option?