Unable to create a partitioned table on s3 data

Constantine · ‎04-10-2022

I write data to s3 like

data.write.format("delta").mode("append").option("mergeSchema", "true").save(s3_location)

and create a partitioned table like

CREATE TABLE IF NOT EXISTS demo_table
USING DELTA
PARTITIONED BY (column_a)
LOCATION {s3_location};

which throws an error

Error in SQL statement: AnalysisException: It is not allowed to specify partitioning when the table schema is not defined.

Is there a different way to write to s3 location so that I can solve this issue

Hubert-Dudek · ‎04-11-2022

@John Constantine , In CREATE TABLE, you need to specify fields:

CREATE TABLE IF NOT EXISTS demo_table
    (column_a STRING,
    number INT)
USING DELTA PARTITIONED BY
     (column_a)
LOCATION 
   {s3_location};

and when you save data before creating a table, it should include a partition. I think it better first register the table and then write to it.

My blog: https://databrickster.medium.com/