cancel
Showing results for 
Search instead for 
Did you mean: 
Community Discussions
cancel
Showing results for 
Search instead for 
Did you mean: 

vscode python project for development

Alexandru
New Contributor III

Hi,

I'm trying to set up a local development environment using python / vscode / poetry. Also, linting is enabled (Microsoft pylance extension) and the python.analysis.typeCheckingMode is set to strict.

We are using python files for our code (.py) whith the "# Databricks notebook source" comment at the begin. That kind of files are handled as notebooks by Databrick

I don't want to connect and execute databricks command from the terminal of vscode.

The question is: which modules shall I import in order to have a valid python file, withouth vscode "problems".

For instance: 

from databricks.sdk import runtime as KR
histogram = KR.spark.sql("some sql")
will raise a the following problem: 
Type of "sql" is partially unknown
Type of "sql" is "(sqlQuery: str, args: Dict[str, Any] | List[Unknown] | None = None, **kwargs: Any) -> DataFrame"PylancereportUnknownMemberType
(variable) spark: SparkSession
Regards,
Alexandru Stan

 

1 ACCEPTED SOLUTION

Accepted Solutions

Alexandru
New Contributor III

After some testing, these are the used modules.

[tool.poetry.group.dev.dependencies]
autopep8 = "^2.1.0"
mypy = "^1.9.0"
pyspark = "=3.5.1"
databricks-sdk = "^0.25.1"
databricks-connect = "^14.3.1"

 Ignore the autopep8, which is only for code quality.

mypy is used only because installs the stubgen utility, which is used for some internal missig stubs
pyspark for the standard pyspark functionality
databricks-sdk for the runtime
databricks-connect extends the sdk

Usind this modules I can set the type checkind mode to strict without any VSCode problems being raised

View solution in original post

3 REPLIES 3

artsheiko
Valued Contributor III
Valued Contributor III

Hi Alexandru,

Take a look at VSCode extension for Databricks : https://marketplace.visualstudio.com/items?itemName=databricks.databricks 

Alexandru
New Contributor III

Sorry @artsheiko ,

as I already said, I'm not interesting in connecting to databricks, I want just to have a valid Python workspace, when using vscode. As a matter of fact, using the extension is not possible in my projects, because the clusters are not interactive and we are not able to use PAT (blocked by our security team)

Regards,

Alex

Alexandru
New Contributor III

After some testing, these are the used modules.

[tool.poetry.group.dev.dependencies]
autopep8 = "^2.1.0"
mypy = "^1.9.0"
pyspark = "=3.5.1"
databricks-sdk = "^0.25.1"
databricks-connect = "^14.3.1"

 Ignore the autopep8, which is only for code quality.

mypy is used only because installs the stubgen utility, which is used for some internal missig stubs
pyspark for the standard pyspark functionality
databricks-sdk for the runtime
databricks-connect extends the sdk

Usind this modules I can set the type checkind mode to strict without any VSCode problems being raised