Databricks Community

KKDataEngineer · 01-28-2022

Team, I am struggling with a unique issue. I am not sure if my understanding is wrong or this is a bug with spark. I am reading a stream from events hub ( Extract) Pivoting and Aggregating the above dataframe ( Transformation). This is a WATERMARKED...

KKDataEngineer · 06-30-2023

No we have not done any custom library configs. You could be right on the limitations.

KKDataEngineer · 06-30-2023

@karthik_p we are still in the preliminary steps on this in our SAAS product. we have not thoroughly tested but have not seen any issues so far. we were able to consistently add libraries to all the clusters with the init scrip in a UC enable worksp...

KKDataEngineer · 06-30-2023

@ah0896 it worked when using a workspace level shell scipt file and pointing to its path with workspace scope.

KKDataEngineer · 06-30-2023

@Retired_mod I was able to solve this Add this code into a simple scala class object method package it into a JAR filenow install this JAR file on the cluster where you execute the JDBC code.add the below line of code before executing the JDBC code...

KKDataEngineer · 06-28-2023

is there a solution for this?

Databricks Community

User Stats

User Activity

Spark Structred Streaming, An Aggregation DF with Watermark in Append mode to Delta table is not writing the most recent aggregation to the Delta table even after crossing the water mark boundary. This is causing dataloss

Re: Using init scripts on UC enabled shared access mode clusters

Re: Using init scripts on UC enabled shared access mode clusters

Re: Using init scripts on UC enabled shared access mode clusters

Re: How to register a JDBC Spark dialect in Python?

Re: How to register a JDBC Spark dialect in Python?