Unable to create a partitioned table on s3 data

Constantine — Mon, 11 Apr 2022 05:56:12 GMT

I write data to s3 like

data.write.format("delta").mode("append").option("mergeSchema", "true").save(s3_location)

and create a partitioned table like

CREATE TABLE IF NOT EXISTS demo_table
USING DELTA
PARTITIONED BY (column_a)
LOCATION {s3_location};

which throws an error

Error in SQL statement: AnalysisException: It is not allowed to specify partitioning when the table schema is not defined.

Is there a different way to write to s3 location so that I can solve this issue

Re: Unable to create a partitioned table on s3 data

Hubert-Dudek — Mon, 11 Apr 2022 20:10:21 GMT

@John Constantine , In CREATE TABLE, you need to specify fields:

CREATE TABLE IF NOT EXISTS demo_table
    (column_a STRING,
    number INT)
USING DELTA PARTITIONED BY
     (column_a)
LOCATION 
   {s3_location};

and when you save data before creating a table, it should include a partition. I think it better first register the table and then write to it.