topic Re: Unable to fetch 3rd level array using dot notation from a json data in local spark via VS Code in Data Engineering

Unable to fetch 3rd level array using dot notation from a json data in local spark via VS Code

Ritasree — Mon, 21 Jul 2025 10:33:06 GMT

I am able to fetch 3rd level array using dot notation from a json in Databricks but same code is not working in local spark via VS Code.

Example - df.select(F.col('x.y.z')) where x is array, y is array and z is also an array.

In local spark I am getting error parameter type mismatch.

Re: Unable to fetch 3rd level array using dot notation from a json data in local spark via VS Code

szymon_dybczak — Mon, 21 Jul 2025 11:30:35 GMT

Hi @Ritasree ,

When you're saying about local spark you mean that you've configured it locally, or are you using Databricks Connect? If you're configured spark locally, then check what version of Spark you're using.

Re: Unable to fetch 3rd level array using dot notation from a json data in local spark via VS Code

Ritasree — Wed, 30 Jul 2025 06:13:59 GMT

I have configured it locally and Spark version is 3.5.1