Databricks job getting stuck at saveAsTable

bhakti
New Contributor II

‎08-28-2024 12:50 AM

I am trying to write logs to delta table,  but after running for sometime the job is getting stuck at saveAsTable.


Traceback (most recent call last):
File "/databricks/spark/python/pyspark/errors/exceptions.py", line 228, in deco
return f(*a, **kw)
File "/databricks/spark/python/lib/py4j-0.10.9.5-src.zip/py4j/protocol.py", line 326, in get_return_value
raise Py4JJavaError(
py4j.protocol.Py4JJavaError: An error occurred while calling o3007.saveAsTable.

Any suggestions would be helpful.

0 Kudos

Rishabh-Pandey
Databricks MVP

‎08-28-2024 01:29 AM

@bhakti looks like some permission issues is there . can you please share the code to take more reference.

Rishabh Pandey
0 Kudos

bhakti
New Contributor II
In response to Rishabh-Pandey

‎08-28-2024 02:16 AM

I am using delta format to save the logs into a table as below. df.write.format("delta").mode("append").saveAsTable("table_name")
0 Kudos

Rishabh-Pandey
Databricks MVP
In response to bhakti

‎08-28-2024 03:17 AM

@bhakti  @RohitKulkarni  Both can be the reason either the permission issue or schema mismatch .

Rishabh Pandey
0 Kudos

RohitKulkarni
Contributor II
In response to bhakti

‎08-28-2024 03:37 AM

@bhakti: Please run the below script :

try:
df.write.format("delta").mode("append").saveAsTable("table_name")
except Exception as e:
print(f"Error: {str(e)}")

and let us know the error.

Thanks

Rohit

 

0 Kudos

RohitKulkarni
Contributor II

‎08-28-2024 02:04 AM

@bhakti : There is issue with data type mis-match.

0 Kudos

RohitKulkarni
Contributor II

‎08-28-2024 02:06 AM

@bhakti : These is related to Data type mis-match

 

Regards

Rohit

0 Kudos

ivanychev
Contributor II

‎08-28-2024 04:12 AM

Hey @bhakti ! Please provide the full stack trace / error message. Your log doesn't provide any strong clue, the failure during write might occur for various reasons. 

Sergey
0 Kudos