ReduceFunction mechanism

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

ReduceFunction mechanism

nragon
Hi,

Regarding ReduceFunction.
Is reduce() called when there is only one record for a given key?

Thanks
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: ReduceFunction mechanism

Fabian Hueske-2
No, the reduce() method of a ReduceFunction requires two elements.
The first received element is just put into state. Once the second element arrives, both are given to the ReduceFunction and the result is put into state and replaces the first element.

Best, Fabian

2017-06-12 18:25 GMT+02:00 nragon <[hidden email]>:
Hi,

Regarding ReduceFunction.
Is reduce() called when there is only one record for a given key?

Thanks



--
View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/ReduceFunction-mechanism-tp13651.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: ReduceFunction mechanism

nragon
So, if my reduce function applies some transformation I must migrate that transformation to a map before the reduce to ensure it transforms, even if there is only one element?
I can chain them together and it will be "almost" as they were in the same function(Ensure same thread processing)?
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: ReduceFunction mechanism

Fabian Hueske-2
You can use a MapFunction (however, it will touch each element and not only the first).
An alternative could be the AggregateFunction if you are using ReduceFunction on a WindowedStream. The interface is a bit more complex though.

Best, Fabian



2017-06-13 10:55 GMT+02:00 nragon <[hidden email]>:
So, if my reduce function applies some transformation I must migrate that
transformation to a map before the reduce to ensure it transforms, even if
there is only one element?
I can chain them together and it will be "almost" as they were in the same
function(Ensure same thread processing)?



--
View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/ReduceFunction-mechanism-tp13651p13679.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: ReduceFunction mechanism

nragon
My goal is to touch each element before the aggregation but i could do it in the reduce function a not having to add another function, thus creating more overhead. The reduce method receives the reduced and a new element which i would change and apply my aggregation.
I'm doing keyby->reduce.
Using a map before all this is a solution.
I've  never tried AggregateFunction, any examples?

Thanks,
Nuno
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: ReduceFunction mechanism

nragon
just an fyi: currently i'm doing map -> keyby -> reduce which in fact could only be keyby -> reduce since reduce can have the map logic.
Loading...