cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

Disable spark transformations inside try except warning

Kayla
Valued Contributor II
I'm wondering if there's a way to suppress certain errors? I have a lot of spammy yellow underlining from this one warning, and it doesn't seem that the warning behaves properly. Transformation inside try/except block is lazy, fine, I have actions on the dataframe ALSO in the try/except so this is irrelevant.

I'm also a little confused what it's actually coming from. It says it's from databricks-flint, which sounds like a linter, but searching it is giving me a time-series library?

Warning message:
Spark transformations inside try/except are lazy and will not raise exceptions. Move the triggering action (e.g., .count(), .show(), .collect()) inside the try block instead.
databricks-flint(SCPAP005)
 
Kayla_0-1785436282119.png

 

 
4 REPLIES 4

adnan_alvee
Databricks Employee
Databricks Employee

Can u try this in your toml file?

[tool.databricks.flint]
ignore = ["SCPAP005"]

Kayla
Valued Contributor II

So...I don't have one of those? Guessing you mean pyproject.toml which I just added with your suggestion, and no change.

masonreed11
New Contributor II

If your action (count(), collect(), show(), etc.) is already inside the try block, this sounds like a false positive. The warning is likely from a Databricks inspection rather than the Flint time-series library. Check if inspection SCPAP005 can be disabled or its severity changed in your IDE, and if not, consider reporting it with a small reproducible example so the maintainers can investigate.

Kayla
Valued Contributor II

I'm seeing this in the Databricks web editor, not my own IDE environment.
I'd disable it if I knew how, but I can't find any documentation for what it's coming from. If you google SCPAP005, all you get is this post and CPAP machines. Likewise, I'd report it to the maintainers if I knew who they were.