Databricks

Sandesh87 · 06-13-2023

The dataframe 'big_df' looks like the below| id| index| timestamp||:---- |:------:| -----:|| abc| 1| 11:00:00|| abc| 1| 11:00:10|| abc| 1| 11:00:20|| abc| 1| 11:00:30|| abc| 1| 11:00:40|| abc| 1| 11:00:50|| abc| 2| 11:01:00|| abc| 2| 11:01:10|| abc| ...

Sandesh87 · 03-08-2023

An application receives messages from event hub. Below is a message received from event hub and loaded into a dataframe with one columnname,gender,idsam,m,001-----time,x,y,z,long,lat160,22,45,51,83,56230,82,95,48,18,26-----event,a,b,c034,1,5,6073,4,2...

Sandesh87 · 12-08-2022

The azure event hub "my_event_hub" has a total of 5 partitions ("0", "1", "2", "3", "4")The readstream should only read events from partitions "0" and "4"event hub configuration as streaming source:-val name = "my_event_hub" val connectionString = "m...

Sandesh87 · 05-27-2022

I have a getS3Object function to get (json) objects located in aws s3 object client_connect extends Serializable { val s3_get_path = "/dbfs/mnt/s3response" def getS3Objects(s3ObjectName: String, s3Client: AmazonS3): String = { val...

Sandesh87 · 03-08-2022

I am trying to execute an api call to get an object(json) from amazon s3 and I am using foreachPartition to execute multiple calls in paralleldf.rdd.foreachPartition(partition => { //Initialize list buffer var buffer_accounts1 = new ListBuffer[St...

Sandesh87 · 03-08-2023

appreciate the feedback but i cant control what is coming through event hub. The message is just the way it is and cant be undone

Sandesh87 · 12-19-2022

@Uma Maheswara Rao Desula Do you mean creating a config map such that that the offset (event position) of the start and end event positions on the partitions are the same?Example1: In both starting and ending positions:-EventPosition.fromOffset("1")...

Sandesh87 · 03-08-2022

It can be achieved using mapPartitionsval df_response = df.mapPartitions(iterator => { val api_connect = new s3clientBuild() val s3client = api_connect.s3connection(AccessKey, SecretKey) val res = iterator.map(row =>{ val name = getS3(row....

Databricks

User Stats

User Activity

apply a function across multiple smaller dataframes created from one big dataframe in scala

parse and combine multiple datasets within a single file

spark-streaming read from specific event hub partition

Task not serializable: java.io.NotSerializableException: org.apache.spark.sql.streaming.DataStreamWriter

create a dataframe with all the responses from the api requests within foreachPartition

Re: parse and combine multiple datasets within a single file

Re: spark-streaming read from specific event hub partition

Re: create a dataframe with all the responses from the api requests within foreachPartition