Accumulating a batch

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Accumulating a batch

Austin Cawley-Edwards
Hi there,

Is it possible to use an AggregationFunction to accumulate n values in a buffer until a threshold is met, then stream down all records in the batch?

Thank you!
Austin Cawley-Edwards
Reply | Threaded
Open this post in threaded view
|

Re: Accumulating a batch

Hequn Cheng
Hi Austin,

You can use GroupBy Window[1], such as TUMBLE Window. The size of the window either as time or row-count interval. You can also define your own User-Defined Aggregate Functions[2] to be used in window.

Best, Hequn


On Fri, Oct 26, 2018 at 5:08 AM Austin Cawley-Edwards <[hidden email]> wrote:
Hi there,

Is it possible to use an AggregationFunction to accumulate n values in a buffer until a threshold is met, then stream down all records in the batch?

Thank you!
Austin Cawley-Edwards
Reply | Threaded
Open this post in threaded view
|

Re: Accumulating a batch

Austin Cawley-Edwards
Hi Hequn,

Thank you for the info! Much appreciated! Do you know if a GroupBy Window could be used to buffer until an arbitrary record came through?

Best,
Austin

On Thu, Oct 25, 2018, 9:17 PM Hequn Cheng <[hidden email]> wrote:
Hi Austin,

You can use GroupBy Window[1], such as TUMBLE Window. The size of the window either as time or row-count interval. You can also define your own User-Defined Aggregate Functions[2] to be used in window.

Best, Hequn


On Fri, Oct 26, 2018 at 5:08 AM Austin Cawley-Edwards <[hidden email]> wrote:
Hi there,

Is it possible to use an AggregationFunction to accumulate n values in a buffer until a threshold is met, then stream down all records in the batch?

Thank you!
Austin Cawley-Edwards