Databricks

Soma · 02-17-2024

node_timeline system table is not producing any output in UC.In system tables node_timeline table is not having any data .Do we need to enable to any other services for UC

Soma · 12-09-2023

We are trying to capture the query executed by spark .We are trying to use df.queryExecution.redactedSql to get the SQL from query execution but it is not working in sqlListener

Soma · 08-08-2023

We use pyspark streaming listener and it is lagging for 10 hrsThe data streamed in 10 am IST is logged at 10 PM IstCan someone explain how logging listener interface work

Soma · 06-18-2023

I used rocksdb statestore for streaming and use collect_set to achieve de-dup logic.Is there any way I can convert rocksdb key value iterator to normal string as I need to validate how internally it is stored as I might need to store 50k distinct val...

Soma · 04-23-2023

Databricks Workflow cost on running in interactive cluster

Soma · 08-08-2023

No spark runs in UTC and log analytics or python modules also uses UTC.One more key point is even in dev we had the same code but we had only a latency<2 mins.Something potentially wrong with pyspark listener

Soma · 08-08-2023

I have added below timestampmicrobatch_timestamp --> which is coming from stream progress and it is showing 10 ambatchid -> which is the spark batchid of stream is also corresponding to 10 amwhere as ingested_timestamp which is showing around10 pm w...

Soma · 08-08-2023

we use Azure log analytics and in log analytics we don't see a lag aswe capture current_timestamp value in (inprogress) and log analytics timestamp_Generated(automatic column)which get added when data is written are in difference of 1 min and progre...

Soma · 08-08-2023

hi @willam45 the spark streaming application is working perfectly as I could see from structured streaming UIbut this custom listener which I use to log to alert real time has some issues .I just wanted to know how it is handled asynchronously by s...

Soma · 04-24-2023

DLT live tables are costlier

Databricks

User Stats

User Activity

node_timeline system table is not producing any output in UC

df.queryExecution.redactedSql is not working with Spark sql Listener

spark streaming listener is lagging

View rocksdb statestore

Databricks Workflow cost on running in interactive cluster

Re: spark streaming listener is lagging

Re: spark streaming listener is lagging

Re: spark streaming listener is lagging

Re: spark streaming listener is lagging

Re: Databricks Workflow cost on running in interactive cluster