Glue database and saveAsTable
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
08-06-2024 08:23 AM
Hello all,
I am saving my data frame as a Delta Table to S3 and AWS Glue using pyspark and `saveAsTable`, so far I can do this but something curious happens when I try to change the `path` (as an option or as an argument of `saveAsTable`).
The location in my Glue table is not updated to the correct path, instead it adds the suffix __PLACEHOLDER__, for example if I want to save the data frame as `my_table` in a bucket `s3://my-bucket/data/my_table` on the Glue table I will see the location as `s3://my-bucket/my_table-__PLACEHOLDER__`. However I still can query my table through SQL or pyspark.
My current workaround is to save the table and next to update the location on Glue using boto3.
Do you know if it is possible to make `saveAsTable` work as expected ? Or do you have another workaround ?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
08-08-2024 03:55 AM
Hi @mexcram, When saving a DataFrame as a Delta Table to S3 and AWS Glue using PySpark's `saveAsTable`, changing the `path` option or argument often results in the Glue table location being set to a placeholder path (e.g., `s3://my-bucket/my_table-__PLACEHOLDER__`), even though you can still query the table. This issue occurs because the `path` change is not properly reflected in Glue's metadata. The current workaround is to manually update the location using boto3 after saving the table. Unfortunately, as of now, there's no built-in way to directly make `saveAsTable` update the path as expected, so continuing with the boto3 workaround is advisable.
If you need further assistance, feel free to ask!
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
08-08-2024 05:11 AM
Thank you, I will continue doing this.
