- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
08-08-2024 03:55 AM
Hi @mexcram, When saving a DataFrame as a Delta Table to S3 and AWS Glue using PySpark's `saveAsTable`, changing the `path` option or argument often results in the Glue table location being set to a placeholder path (e.g., `s3://my-bucket/my_table-__PLACEHOLDER__`), even though you can still query the table. This issue occurs because the `path` change is not properly reflected in Glue's metadata. The current workaround is to manually update the location using boto3 after saving the table. Unfortunately, as of now, there's no built-in way to directly make `saveAsTable` update the path as expected, so continuing with the boto3 workaround is advisable.
If you need further assistance, feel free to ask!