Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
08-08-2023 07:19 AM
I have added below timestamp
microbatch_timestamp --> which is coming from stream progress and it is showing 10 am
batchid -> which is the spark batchid of stream is also corresponding to 10 am
where as ingested_timestamp which is showing around10 pm which is when it is loaded to log analytics
def onQueryProgress(self, queryProgress):
try:
print("inside on progress")
# Write the input rate and processing time to ADLS
if(queryProgress.progress.numInputRows >= 0😞
progress = queryProgress.progress
data = {
"Stream_id" : progress.id.__str__() ,
"Stream_name" : progress.name,
"Event_type" : "Query Progress",
"runID" : progress.runId.__str__(),
"microbatch_timestamp" : progress.timestamp.split('.')[0],
"ingested_timestamp" : datetime.now().strftime("%Y-%m-%d %H:%M:%S.%f"),
"batchId" : progress.batchId,
