Databricks Community

Rishi045 · 08-03-2023

Hi All,I am facing an issue of data getting missed.I am reading the data from azure event hub and after flattening the json data I am storing it in a parquet file and then using another databricks notebook to perform the merge operations on my delta ...

Rishi045 · 08-03-2023

As of now I am not having any foreachbatch in my code. I am performing dedup on entire data coming from event hub

Rishi045 · 08-03-2023

in event hub i can see the missing records. should I be using forEachBatch somewhere in my code. ?or any other suggestion.

Rishi045 · 08-03-2023

I am dropping duplicates based on my primary key as multiple records are coming from event hub for a single primary key and I want to take the latest one also i am doing order by on op_ts column so that I get the latest record only.Distinct can creat...

Rishi045 · 08-03-2023

Using below code :conf = {}df = spark.readStream.format("eventhubs").options(**conf).load()dataDF = df.select(col("body").cast("STRING"))data = dataDF.select(json_tuple(col("body"),"table","op_type","records","op_ts")) \ .toDF("table","op_...

Rishi045 · 08-03-2023

I could not find those missing records in data lake as well

Databricks Community

User Stats

User Activity

Data getting missed while reading from azure event hub using spark streaming

Re: Data getting missed while reading from azure event hub using spark streaming

Re: Data getting missed while reading from azure event hub using spark streaming

Re: Data getting missed while reading from azure event hub using spark streaming

Re: Data getting missed while reading from azure event hub using spark streaming

Re: Data getting missed while reading from azure event hub using spark streaming