cancel
Showing results for 
Search instead for 
Did you mean: 
Get Started Discussions
Start your journey with Databricks by joining discussions on getting started guides, tutorials, and introductory topics. Connect with beginners and experts alike to kickstart your Databricks experience.
cancel
Showing results for 
Search instead for 
Did you mean: 

Space in Column names when writing to Hive

JohnJustus
New Contributor III

All,

I have the following code.
df_Warehouse_Utilization = (

    spark.table("hive_metastore.dev_ork.bin_item_detail")
    .join(df_DIM_Bins,col('bin_tag')==df_DIM_Bins.BinKey,'right')
    .groupby(col('BinKey'))
    .agg(count_distinct(when(col('serial_lot_tag')>0,col('serial_lot_tag'))).alias('SerialLotCount'),
         count_distinct(when(col('tracking_tag')>0,col('tracking_tag'))).alias('Track Count'),
          min(col('activation_time')).alias('OldestStorageTime')
    )
    .withColumn('BinUsedFlag',when(col('PartNumbers')>0,1).otherwise(0))

)
df_Warehouse_Utilization.write.mode("overwrite").format("delta").saveAsTable("yorkgold.df_DIM_Binsst")
display(df_Warehouse_Utilization)

when I write to table after the above transformation, the following error occurs.
AnalysisException: Found invalid character(s) among ' ,;{}()\n\t=' in the column names of your schema.

The issue is because I have the space in the column name (Track Count, please see the above highlighted), As soon as I rename column as (TrackCount), I am able to write to Hive.
So my question is, space is not allowed? Or is there any other option?

2 REPLIES 2

daniel_sahal
Esteemed Contributor

@JohnJustus You can try using backticks (`) to escape a column names that contains spaces, but the suggestion here is to get rid of spaces or any special characters in a column names.

Thank you Daniel

So I have my code below with back stick but still doesn't work -   col('bin_type').alias(`'Bin Type'`),

Can you please correct me?

df_Bins_Step1 = (
    spark.table("hive_metastore.dev_fivetranpoc_york.bin_master")
    .select(col('tag_number').alias('BinKey'),
        col('bin').alias('Bin'),
        col('bin_type').alias(`'Bin Type'`),
        col('zone').alias('Zone'))
)
display(df_Bins_Step1)

Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!