Unable to fetch 3rd level array using dot notation from a json data in local spark via VS Code
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
07-21-2025 03:33 AM
I am able to fetch 3rd level array using dot notation from a json in Databricks but same code is not working in local spark via VS Code.
Example - df.select(F.col('x.y.z')) where x is array, y is array and z is also an array.
In local spark I am getting error parameter type mismatch.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
07-21-2025 04:30 AM
Hi @Ritasree ,
When you're saying about local spark you mean that you've configured it locally, or are you using Databricks Connect? If you're configured spark locally, then check what version of Spark you're using.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
07-29-2025 11:13 PM
I have configured it locally and Spark version is 3.5.1
