Databricks Community

pranathisg97 · 06-06-2023

I want to execute this statement using databricks SQL Statement Execution API. curl -X POST -H 'Authorization: Bearer <access-token>' -H 'Content-Type: application/json' -d '{"warehouse_id": "<warehouse_id>", "statement": "set us...

pranathisg97 · 03-02-2023

Is it normal for KinesisSource to generate empty microbatches when there is no new data in Kinesis? Batch 1 finished as there were records in kinesis and BatchId 2 started. BatchId 2 was running but then BatchId 3 started . Even though there was no m...

pranathisg97 · 02-24-2023

Hi,I have a scenario where writeStream query writes the stream data to bronze location and I have to read from bronze, do some processing and finally write it to silver. I use S3 location for delta tablesBut for the very first execution , readStream ...

pranathisg97 · 02-15-2023

I want to fetch new data from kinesis source for every minute. I'm using "minFetchPeriod" option and specified 60s. But this doesn't seem to be working.Streaming query: spark \ .readStream \ .format("kinesis") \ .option("streamName", kinesis_stream_...

pranathisg97 · 02-24-2023

Hi @Jose Gonzalez ,Thanks for sharing the document. I would like to share my understanding from the document. Please correct me if I'm wrong.minFetchPeriod is the time interval between consecutive pre-fetch between kinesis and spark. maxFetchDuratio...

pranathisg97 · 02-21-2023

Hi,I was able to solve the issue by specifying maxFetchDuration value more than minFetchPeriod. However I didn't get the purpose of specifying maxFetchDuration value greater than or equal to minFetchPeriod value.spark \ .readStream \ .format("kinesis...

pranathisg97 · 02-17-2023

Hi, DBR is 12.1 Error : "java.lang.IllegalArgumentException: maxFetchDuration needs to be a duration string such as '10s' for 10 seconds or '2m' for 2 minutes. Also needs to be greater than minFetchPeriod in order to avoid falling behind in the strea...

Databricks Community

User Stats

User Activity

Control query caching using SQL statement execution API

KinesisSource generates empty microbatches when there is no new data.

readStream query throws exception if there's no data in delta location.

Fetch new data from kinesis for every minute.

Re: Fetch new data from kinesis for every minute.

Re: Fetch new data from kinesis for every minute.

Re: Fetch new data from kinesis for every minute.