- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
08-26-2022 08:32 AM
I have renamed a table, however on trying to write to it (or delete from it) I get the following error:
`java.io.FileNotFoundException: No such file or directory: s3a://.../hive/warehouse/testing.db/renamed_table_name/_delta_log/00000000000000000002.json`
My cluster version is 10.4 LTS
The following python snippet can be used to recreate the issue:
```
DB_NAME = 'testing'
def rename(old_table_name, new_table_name):
spark.conf.set("spark.databricks.delta.alterTable.rename.enabledOnAWS", "true")
spark.sql(f"ALTER TABLE {DB_NAME}.{old_table_name} RENAME TO {DB_NAME}.{new_table_name}")
def overwrite(table_name, source_df):
(source_df.write
.mode("overwrite")
.format("delta")
.saveAsTable(f"{DB_NAME}.{table_name}")
)
def get_df():
data = [
("Name1", "Surname1", 12345),
("Name2", "Surname2", 67890),
]
return spark.createDataFrame(data)
sparkDf = get_df()
# Create table
overwrite('table_name', sparkDf)
# Rename table
rename('table_name', 'renamed_table_name')
# Try to write to renamed table (or delete from). Failure happens here
overwrite('renamed_table_name', sparkDf)
```
Any help would be appreciated
Accepted Solutions
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
08-31-2022 06:16 AM
I tried your code in 11.1 (includes Apache Spark 3.3.0, Scala 2.12) and it worked perfectly fine.
However it will work perfectly only the first time because if you try to rerun the same code, it'll error out stating renamed_table_name exists.
P.S: Please use code snippet(</> button) to include your code in your question
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
08-31-2022 06:16 AM
I tried your code in 11.1 (includes Apache Spark 3.3.0, Scala 2.12) and it worked perfectly fine.
However it will work perfectly only the first time because if you try to rerun the same code, it'll error out stating renamed_table_name exists.
P.S: Please use code snippet(</> button) to include your code in your question
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-15-2022 03:36 AM
Hi @julie.holtzhausen
Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help.
We'd love to hear from you.
Thanks!
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-30-2022 04:19 AM
Hi @477061 Could you please try to test it in DBR 11.1 and see if the issue persists for you?