cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

"Fatal error: The Python kernel is unresponsive." DBR 14.3

ac0
New Contributor III

Running almost any notebook with a merge statement in Databricks with DBR 14.3 I get the following error and the notebook exists:

"Fatal error: The Python kernel is unresponsive."

I would provide more code, but like I said, it is pretty much anything with a merge statement, except for the tiniest of batches. Anything larger than that fails. Changing the compute to DBR 13.3 resolves the issue. Has anyone else encountered anything similar?

2 REPLIES 2

Ayushi_Suthar
Honored Contributor
Honored Contributor

Hi @ac0 , Hope you are doing well! 

The "Fatal error: The Python kernel is unresponsive" error generally means the Jupyter Kernel is still running but is not responsive within a specific time frame.

Please try to increase the timeout by specifying config "spark.databricks.driver.python.pythonHealthCheckTimeoutSec" to 300 (5 min, default is 1.5 min). And let us know if you still see the error after applying the configuration.

Also we do have a kb article around this issue explaining the cause and possible solution to resolve the issue in future.
https://kb.databricks.com/en_US/clusters/python-kernel-is-unresponsive-error-message

Please let me know if this helps and leave a like if this information is useful, followups are appreciated.
Kudos
Ayushi

ac0
New Contributor III

Thanks @Ayushi_Suthar. I will try this and report back. Is "spark.databricks.driver.python.pythonHealthCheckTimeoutSec" a new configuration setting? Literally nothing on Google or Stackoverflow is returned when I search by either that full name or even just "pythonHealthCheckTimeoutSec." I want to make sure I have a good understanding of it, so is there any documentation or further information you have that I could review?

Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!