topic Re: Writing PySpark DataFrame onto AWS Glue throwing error in Data Engineering

Writing PySpark DataFrame onto AWS Glue throwing error

raghub1 — Tue, 10 May 2022 08:12:02 GMT

I have followed the steps as mentioned in this blog : https://www.linkedin.com/pulse/aws-glue-data-catalog-metastore-databricks-deepak-rajak/

but when trying to saveAsTable(table_name), it is giving an error as

IllegalArgumentException: Path must be absolute: <table_name>-__PLACEHOLDER__.

Can somebody help me on this please ?

Re: Writing PySpark DataFrame onto AWS Glue throwing error

Prabakar — Thu, 12 May 2022 12:55:48 GMT

Looking at the error message I believe the problem is the Glue database has no location which DBR/Delta needs. You can use alter database datalake-processed set location='...' or set the location directly in Glue console on AWS.

https://docs.databricks.com/data/metastores/aws-glue-metastore.html#creating-a-table-in-a-database-with-empty-location

Re: Writing PySpark DataFrame onto AWS Glue throwing error

raghub1 — Thu, 12 May 2022 13:32:14 GMT

Thanks Prabhakar, I used the option of specifying the path as s3, but it wouldn't work. This is the code I used :

df_final.write.partitionBy("partition_cols").mode("append").option("path", "s3:// location").saveAsTable("table_name")

Can you help me out please ?

Re: Writing PySpark DataFrame onto AWS Glue throwing error

Anonymous — Tue, 21 Jun 2022 16:10:24 GMT

Hey @Raghu Bharadwaj Tallapragada

Just wanted to check in if you were able to resolve your issue or do you need more help? We'd love to hear from you.

Thanks!

Re: Writing PySpark DataFrame onto AWS Glue throwing error

prabhatika — Sat, 04 Mar 2023 06:20:11 GMT

@Kaniz Fatma

I am also facing the same issue while using the `saveAsTable` function of DataFrameWriter. Following is the code snippet: -

import org.apache.spark.sql.functions.{col, dayofmonth, month, to_date, year}
import org.apache.spark.sql.types.DataTypes
 
val df = some-dataframe-here
val glueTableName = "database-name-here.table-name-here"
val s3Path = "s3a://some/path/here/"
val partitionKeys = Array("some-partition-key-here")
val dataframeWithYearMonthDay = df
                                  .withColumn("year", year(to_date(col("createdAt"))).cast(DataTypes.FloatType))
                                  .withColumn("month", month(to_date(col("createdAt"))).cast(DataTypes.FloatType))
                                  .withColumn("day", dayofmonth(to_date(col("createdAt"))).cast(DataTypes.FloatType))
dataframeWithYearMonthDay.write
.partitionBy(List("year", "month", "day") ++ partitionKeys: _*)
.mode("append")
.format("parquet")
.option("path", s3Path)
.saveAsTable(glueTableName)

PFA the stack trace. Please note that the given s3 location is completely empty and I am trying to create a new table here.

Also, I am facing this issue with only one table. Not facing this issue with writing to other tables.

Please let me know if any other information is needed from my end.