topic Pylint github workflow in Data Engineering

Pylint github workflow

carlos_tasayco — Mon, 28 Apr 2025 22:32:24 GMT

I am implementing this to my workflow however there is a problem I want to avoid

Enrollment_Profile/mv_person_test_pylint.py:22:9: E0602: Undefined variable 'spark' (undefined-variable)

Can someone avoid that issue with spark dlt?

Re: Pylint github workflow

samshifflett46 — Fri, 02 May 2025 19:03:06 GMT

Hey @carlos_tasayco, it seems like you are running a test here. If you are running a test outside of a databricks environment (like in a CI pipeline) then you need to define a spark session manually. In the DBR (databricks runtime) the spark session is created automatically for you but when you run code locally or in a CI pipeline, then you need to build the spark session yourself and save it in the variable 'spark'.

Check out this thread and look at MarkusFra's reply, he creates a Databricks Session object in his script which is what I think you need to do here.

https://community.databricks.com/t5/data-engineering/re-establish-sparksession-using-databricks-connect-after-cluster/td-p/64394

Re: Pylint github workflow

carlos_tasayco — Fri, 02 May 2025 19:14:30 GMT

Hi,

Yes I found that solution too, the problem after that was dbutils, pylint identify that as a no imported library. I found a solution at least for me in the pylintrc file I added this:

Doing this I avoide this problem.