topic Re: Glue database and saveAsTable in Data Engineering

Glue database and saveAsTable

mexcram — Tue, 06 Aug 2024 15:23:56 GMT

Hello all,

I am saving my data frame as a Delta Table to S3 and AWS Glue using pyspark and `saveAsTable`, so far I can do this but something curious happens when I try to change the `path` (as an option or as an argument of `saveAsTable`).

The location in my Glue table is not updated to the correct path, instead it adds the suffix __PLACEHOLDER__, for example if I want to save the data frame as `my_table` in a bucket `s3://my-bucket/data/my_table` on the Glue table I will see the location as `s3://my-bucket/my_table-__PLACEHOLDER__`. However I still can query my table through SQL or pyspark.

My current workaround is to save the table and next to update the location on Glue using boto3.

Do you know if it is possible to make `saveAsTable` work as expected ? Or do you have another workaround ?

Re: Glue database and saveAsTable

Retired_mod — Thu, 08 Aug 2024 10:55:03 GMT

Hi @mexcram, When saving a DataFrame as a Delta Table to S3 and AWS Glue using PySpark's `saveAsTable`, changing the `path` option or argument often results in the Glue table location being set to a placeholder path (e.g., `s3://my-bucket/my_table-__PLACEHOLDER__`), even though you can still query the table. This issue occurs because the `path` change is not properly reflected in Glue's metadata. The current workaround is to manually update the location using boto3 after saving the table. Unfortunately, as of now, there's no built-in way to directly make `saveAsTable` update the path as expected, so continuing with the boto3 workaround is advisable.

If you need further assistance, feel free to ask!

Re: Glue database and saveAsTable

mexcram — Thu, 08 Aug 2024 12:11:56 GMT

Thank you, I will continue doing this.