Databricks Community

VikasM

I'm working on a personal data engineering project using Kafka, Spark Structured Streaming, and Docker.The application consumes two Kafka topics that originate from an external market-data websocket source:a trade streama candlestick (kline/OHLCV) st...

VikasM

Hello Balajij8,I just wanted to let you know that the issue I posted regarding Spark not writing Parquet files was actually due to my own mistake.I had mounted the data volume only in my Spark job (driver/scheduler) container instead of the Spark wor...

VikasM

Thanks for your reply.I investigated the output directories a bit further before trying another path. If my understanding is correct, the volume mount and read/write permissions do not seem to be the issue in my case. The reason I think this is that ...

VikasM

Hello balajij8,Before trying your suggestions, I decided to inspect the filesystem inside my Spark container once more.I found something that has changed my understanding of the problem. There are no errors being reported by the streaming job, and th...

VikasM

Thank you, balajij8, for your suggestions. I really appreciate your time and guidance.I'll try the different configurations you recommended and investigate further. Once I've tested them, I'll come back and share the results.Thanks again for your hel...

VikasM

I have send reply for this message 5 times already I don't know what is going on here

Databricks Community

User Stats

User Activity

PySpark AnalysisException: Ambiguous reference to field t when parsing nested JSON

Re: PySpark AnalysisException: Ambiguous reference to field t when parsing nested JSON

Re: PySpark AnalysisException: Ambiguous reference to field t when parsing nested JSON

Re: PySpark AnalysisException: Ambiguous reference to field t when parsing nested JSON

Re: PySpark AnalysisException: Ambiguous reference to field t when parsing nested JSON

Re: PySpark AnalysisException: Ambiguous reference to field t when parsing nested JSON