topic Re: vscode python project for development in Get Started Discussions

vscode python project for development

Alexandru — Fri, 12 Apr 2024 11:07:52 GMT

Hi,

I'm trying to set up a local development environment using python / vscode / poetry. Also, linting is enabled (Microsoft pylance extension) and the python.analysis.typeCheckingMode is set to strict.

We are using python files for our code (.py) whith the "# Databricks notebook source" comment at the begin. That kind of files are handled as notebooks by Databrick

I don't want to connect and execute databricks command from the terminal of vscode.

The question is: which modules shall I import in order to have a valid python file, withouth vscode "problems".

For instance:

from databricks.sdk import runtime as KR
histogram = KR.spark.sql("some sql")

will raise a the following problem:

Type of "sql" is partially unknown
Type of "sql" is "(sqlQuery: str, args: Dict[str, Any] | List[Unknown] | None = None, **kwargs: Any) -> DataFrame"PylancereportUnknownMemberType
(variable) spark: SparkSession

Regards,

Alexandru Stan

Re: vscode python project for development

artsheiko — Tue, 16 Apr 2024 12:37:50 GMT

Hi Alexandru,

Take a look at VSCode extension for Databricks : https://marketplace.visualstudio.com/items?itemName=databricks.databricks

Re: vscode python project for development

Alexandru — Wed, 17 Apr 2024 06:11:21 GMT

Sorry @artsheiko ,

as I already said, I'm not interesting in connecting to databricks, I want just to have a valid Python workspace, when using vscode. As a matter of fact, using the extension is not possible in my projects, because the clusters are not interactive and we are not able to use PAT (blocked by our security team)

Regards,

Alex

Re: vscode python project for development

Alexandru — Wed, 17 Apr 2024 14:06:43 GMT

After some testing, these are the used modules.

[tool.poetry.group.dev.dependencies] autopep8 = "^2.1.0" mypy = "^1.9.0" pyspark = "=3.5.1" databricks-sdk = "^0.25.1" databricks-connect = "^14.3.1"

Ignore the autopep8, which is only for code quality.

mypy is used only because installs the stubgen utility, which is used for some internal missig stubs
pyspark for the standard pyspark functionality
databricks-sdk for the runtime
databricks-connect extends the sdk

Usind this modules I can set the type checkind mode to strict without any VSCode problems being raised

Re: vscode python project for development

A_N — Fri, 11 Jul 2025 15:10:14 GMT

How did you solve the type error checks on `pyspark.sql ` ?

mypy doesn't create the missing stubs for that one?