topic Re: Databricks job getting stuck at saveAsTable in Data Engineering

Databricks job getting stuck at saveAsTable

bhakti — Wed, 28 Aug 2024 07:50:56 GMT

I am trying to write logs to delta table, but after running for sometime the job is getting stuck at saveAsTable.

Traceback (most recent call last):
File "/databricks/spark/python/pyspark/errors/exceptions.py", line 228, in deco
return f(*a, **kw)
File "/databricks/spark/python/lib/py4j-0.10.9.5-src.zip/py4j/protocol.py", line 326, in get_return_value
raise Py4JJavaError(
py4j.protocol.Py4JJavaError: An error occurred while calling o3007.saveAsTable.

Any suggestions would be helpful.

Re: Databricks job getting stuck at saveAsTable

Rishabh-Pandey — Wed, 28 Aug 2024 08:29:48 GMT

@bhakti looks like some permission issues is there . can you please share the code to take more reference.

Re: Databricks job getting stuck at saveAsTable

RohitKulkarni — Wed, 28 Aug 2024 09:04:32 GMT

@bhakti : There is issue with data type mis-match.

Re: Databricks job getting stuck at saveAsTable

RohitKulkarni — Wed, 28 Aug 2024 09:06:41 GMT

@bhakti : These is related to Data type mis-match

Regards

Rohit

Re: Databricks job getting stuck at saveAsTable

bhakti — Wed, 28 Aug 2024 09:16:31 GMT

I am using delta format to save the logs into a table as below. df.write.format("delta").mode("append").saveAsTable("table_name")

Re: Databricks job getting stuck at saveAsTable

Rishabh-Pandey — Wed, 28 Aug 2024 10:17:47 GMT

@bhakti @RohitKulkarni Both can be the reason either the permission issue or schema mismatch .

Re: Databricks job getting stuck at saveAsTable

RohitKulkarni — Wed, 28 Aug 2024 10:37:58 GMT

@bhakti: Please run the below script :

try:
df.write.format("delta").mode("append").saveAsTable("table_name")
except Exception as e:
print(f"Error: {str(e)}")

and let us know the error.

Thanks

Rohit

Re: Databricks job getting stuck at saveAsTable

ivanychev — Wed, 28 Aug 2024 11:12:51 GMT

Hey @bhakti ! Please provide the full stack trace / error message. Your log doesn't provide any strong clue, the failure during write might occur for various reasons.