Databricks job getting stuck at saveAsTable

[bhakti]

I am trying to write logs to delta table,  but after running for sometime the job is getting stuck at saveAsTable.

Traceback (most recent call last):
File "/databricks/spark/python/pyspark/errors/exceptions.py", line 228, in deco
return f(*a, **kw)
File "/databricks/spark/python/lib/py4j-0.10.9.5-src.zip/py4j/protocol.py", line 326, in get_return_value
raise Py4JJavaError(
py4j.protocol.Py4JJavaError: An error occurred while calling o3007.saveAsTable.

Any suggestions would be helpful.

[Rishabh-Pandey]

@bhakti looks like some permission issues is there . can you please share the code to take more reference.

[bhakti]

I am using delta format to save the logs into a table as below. df.write.format("delta").mode("append").saveAsTable("table_name")

[Rishabh-Pandey]

@bhakti  @RohitKulkarni  Both can be the reason either the permission issue or schema mismatch .

[RohitKulkarni]

@bhakti: Please run the below script :

try:
df.write.format("delta").mode("append").saveAsTable("table_name")
except Exception as e:
print(f"Error: {str(e)}")

and let us know the error.

Thanks

Rohit

[RohitKulkarni]

@bhakti : There is issue with data type mis-match.

[RohitKulkarni]

@bhakti : These is related to Data type mis-match

Regards

Rohit

[ivanychev]

Hey @bhakti ! Please provide the full stack trace / error message. Your log doesn't provide any strong clue, the failure during write might occur for various reasons.