topic Re: Is Limit input rate Docs not correct? in Data Engineering

Is Limit input rate Docs not correct?

Jcowell — Wed, 04 Dec 2024 22:00:49 GMT

"If you use maxBytesPerTrigger in conjunction with maxFilesPerTrigger, the micro-batch processes data until either the maxFilesPerTrigger or maxBytesPerTrigger limit is reached."

But based on the source code this is not true.

val maxBytesPerTrigger: Option[Long] = parameters.get("maxBytesPerTrigger").map { str => Try(str.toLong).toOption.filter(_ > 0).map(op => if (maxFilesPerTrigger.nonEmpty) { throw new IllegalArgumentException( "Options 'maxFilesPerTrigger' and 'maxBytesPerTrigger' " + "can't be both set at the same time") } else op ).getOrElse { throw new IllegalArgumentException( s"Invalid value '$str' for option 'maxBytesPerTrigger', must be a positive integer") } }

Am i missing something here?

Re: Is Limit input rate Docs not correct?

ozaaditya — Thu, 05 Dec 2024 04:23:47 GMT

In my opinion, the reason for not using both options simultaneously is that the framework would face a logical conflict:

Should it stop reading after the maximum number of files is reached, even if the size limit hasn’t been exceeded?
Or
Should it stop processing when the size limit is reached, even if the file count is below the limit?

Handling these scenarios would add unnecessary complexity to the framework.

Re: Is Limit input rate Docs not correct?

Jcowell — Fri, 06 Dec 2024 11:23:09 GMT

I'm not too concern with whether or not they should be used simultaneously as much as the docs saying somethign to be true when the code shows it is not possible.