Databricks

ThomasKastl · 07-26-2022

tl;dr: A cell that executes purely on the head node stops printed output during execution, but output still shows up in the cluster logs. After execution of the cell, Databricks does not notice the cell is finished and gets stuck. When trying to canc...

ThomasKastl · 08-11-2021

A similar question has already been added, but the reply is very confusing to me. Basically, for automated jobs, I want to log the following information from inside a Python notebook that runs in the job: - What is the cluster configuration (most im...

ThomasKastl · 08-01-2022

Thanks! This one actually seems to solve the problem, so I assume the IPython kernel did the trick. Do you know what was used instead, in versions <11.0? Doesn't seem to say so in the docu...

ThomasKastl · 07-27-2022

Do you mean that it uses Spark even if I don't tell it to, somehow recognizing it? Because I am not using spark at all, I am using the exact same code as on local, and when I check spark jobs on the Databricks machine there is nothing (as I expect......

ThomasKastl · 07-26-2022

Thanks for the follow-up. We resolved this a different way, but the input with the dbutils.secret was helpful.

ThomasKastl · 08-11-2021

All the examples seem to use a .netrc file, but it looks like on the cluster, there is none by default - surely, there is some way to check what the cluster config of the current cluster is without going through lots of authentication hoops?

ThomasKastl · 02-19-2019

I have the exact same question. Databricks seems nice for some initial exploration / playing around but once it goes into professionalization being able to only use notebooks sets severe limits...

Databricks

User Stats

User Activity

Databricks runs cell, but stops output and hangs afterwards.

Calling Databricks API from Databricks notebook

Re: Databricks runs cell, but stops output and hangs afterwards.

Re: Databricks runs cell, but stops output and hangs afterwards.

Re: Calling Databricks API from Databricks notebook

Re: Calling Databricks API from Databricks notebook

Re: How to import local python file in notebook?