Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-07-2023 04:16 AM
So can you get the data in a dataframe in databricks so you can create the table? Are you able to access the data in that table with a select statement? Have you made your own metastore for your external location and tied the workspace to it to enable unity catalog for the workspace? If you can do all those things then you can do
df = spark.sql("select * from <catalog>.<schema>.<table>")
df = df.distinct()
df.write.saveAsTable("<catalog>.<schema>.<table>")I also noticed this was the table you were trying to select from?
'org.apache.hadoop.hive.metastore.model.MTable'Where is the catalog and schema and table in that string? If any of those have a dot in it you need to surround each with backticks (``).
#Ex.
table = "`catalog`.`schema`.`table`"
df = spark.sql(f" select * from {table}")