topic Re: Real-time output missing when using “Upload and Run File” from VS Code in Administration & Architecture

Real-time output missing when using “Upload and Run File” from VS Code

tinodj — Thu, 20 Nov 2025 23:08:23 GMT

I am running Python files on a Databricks cluster using the VS Code Databricks extension, specifically the “Upload and Run File” command.

I cannot get real-time output in the Debug Console. I have checked the official docs:

https://learn.microsoft.com/en-us/azure/databricks/dev-tools/vscode-ext/tutorial

https://github.com/databricks/databricks-vscode/blob/release-v2.10.3/packages/databricks-vscode/DATABRICKS.quickstart.md#running-pxspark-code

but these do not address the issue.

The behavior I see is:

While the script is running, no output is shown in the Debug Console.
When the script finishes, all output appears at once.
If the script fails with an exception, only the error is shown and none of the printed output appears.

This makes it very difficult to test and debug. If this is expected behavior, I would like to know what the recommended or best-practice workflow is for running and testing a standalone Python file on a Databricks cluster with live output.

Below is a minimal example based on Databricks’ own sample script. I added a loop with prints and sleeps to demonstrate the missing streaming output:

from pyspark.sql import SparkSession from pyspark.sql.types import * spark = SparkSession.builder.getOrCreate() schema = StructType([ StructField('CustomerID', IntegerType(), False), StructField('FirstName', StringType(), False), StructField('LastName', StringType(), False) ]) data = [ [1000, 'Matthijs', 'Oosterhout-Buntjes'], [1001, 'Joost', 'van Brunswijk'], [1002, 'Stan', 'Bokenkamp'] ] customers = spark.createDataFrame(data, schema) customers.show() import time for i in range(100): print(f'{i}-Output') time.sleep(1) time.sleep(10) raise Exception("Demo failure")

Re: Real-time output missing when using “Upload and Run File” from VS Code

stbjelcevic — Fri, 21 Nov 2025 18:57:20 GMT

Hi @tinodj ,

Are you using the Databricks Connect integration in conjunction with the VS Code extension? I think you should be able to get better debugging results with this enabled.

Re: Real-time output missing when using “Upload and Run File” from VS Code

tinodj — Fri, 21 Nov 2025 19:22:46 GMT

Hi @stbjelcevic

Thanks! With Databricks Connect part of the code runs locally. And this is not always the best scenario to debug the code, or am I wrong? For example when reading some local files mounted on the cluster, or running remote services available only from the cluster, etc. I would like to test when code runs on the cluster in full.

Re: Real-time output missing when using “Upload and Run File” from VS Code

stbjelcevic — Fri, 21 Nov 2025 19:49:21 GMT

Ahh yes, you are right about that. There isn’t a “run all non-Spark code remotely” mode in Databricks Connect today, but it appears to be a commonly requested enhancement and is currently tracked internally.

Have you checked the Databricks UI for the job run to see if the printed outputs show up there? I understand you want local development, but I am trying to figure out if your expected print statements/logs are viewable anywhere to begin with.

Re: Real-time output missing when using “Upload and Run File” from VS Code

tinodj — Fri, 21 Nov 2025 20:50:18 GMT

Yes, prints and loggings are viewable in driver logs as they happen. If the same file is run in databricks Web UI they are viewable on output window as they happen as well. But, when run through VS code, unfortunately they are not visible in the debug console (but they are in the driver logs), as can be demonstrated with the example above.

Re: Real-time output missing when using “Upload and Run File” from VS Code

Dali1 — Wed, 18 Feb 2026 13:00:01 GMT

Hello Any solutions for this ?

Without it it's almost useless since it's not possible to have logs printed out

Re: Real-time output missing when using “Upload and Run File” from VS Code

tinodj — Wed, 18 Feb 2026 21:57:23 GMT

@Dali1 I have even tried to describe https://github.com/databricks/databricks-vscode/issues/1813 and to fix on https://github.com/databricks/databricks-vscode/pull/1814, unfortunately no one seems to be interested in this.