etsyal1e2r3
Honored Contributor

So can you get the data in a dataframe in databricks so you can create the table? Are you able to access the data in that table with a select statement? Have you made your own metastore for your external location and tied the workspace to it to enable unity catalog for the workspace? If you can do all those things then you can do

df = spark.sql("select * from <catalog>.<schema>.<table>")
df = df.distinct()
df.write.saveAsTable("<catalog>.<schema>.<table>")

I also noticed this was the table you were trying to select from?

'org.apache.hadoop.hive.metastore.model.MTable'

Where is the catalog and schema and table in that string? If any of those have a dot in it you need to surround each with backticks (``).

#Ex.
table = "`catalog`.`schema`.`table`"
df = spark.sql(f" select * from {table}")