unable to create delta tables in aws glue catalog
The project requires that we integrate with the AWS Glue catalog.
We would like to be able to create tables in delta format in the glue catalog.
To test this functionality. We did the following
Created the cluster with access to the Glue catalog as described here https://docs.databricks.com/data/metastores/aws-glue-metastore.html
Created a database from AWS Athena
Created a table in Athena based on a csv file in s3
Launched the cluster with glue access in databricks
Ran sql to read table created in step 3 .This worked
Now tried to run a CREATE TABLE AS SELECT .. this failed (error attached.)
I tried to create a parquet table as shown
create table sales222
STORED AS parquet
LOCATION 's3://pla-sc-357559-44-aws-databricks-s3/test/'
as select * from sales_orders
This worked
8 Ran this to convert to delta
convert to delta parquet.`s3://pla-sc-357559-44-aws- databricks-s3/test`
This succeeded. But i noticed that glue catalog ddl still shows format as parquet
Is this the only way to create delta tables on the AWS glue catalog ?
If you can assist with a short demo that would be great
We would like to do this in SQL only as we are integrating with dbt
