Question about the custom partitioner

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Question about the custom partitioner

xingcan
Hi all,

I want to duplicate records to multiple downstream tasks (not all of them thus the
Broadcasting should not work) in stream environment.
However, it seems that the current custom partitioner can return only one partition index.
Why this restriction exists or do I miss something?

Thanks,
Xingcan
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Question about the custom partitioner

Aljoscha Krettek
Hi,

I’m afraid that’s not possible out-of-box with the current APIs. I actually don’t know why the user-facing Partitioner only allows returning one target because the internal StreamPartitioner (which extends ChannelSelector) allows returning multiple target partitions.

You can hack around the API by manually creating your own StreamPartitioner and applying it to the DataStream as DataStream.partitionCustom() and DataStream.setConnectionType() (the first calls the latter) do.

Best,
Aljoscha

On 14. Jun 2017, at 09:09, Xingcan Cui <[hidden email]> wrote:

Hi all,

I want to duplicate records to multiple downstream tasks (not all of them thus the
Broadcasting should not work) in stream environment.
However, it seems that the current custom partitioner can return only one partition index.
Why this restriction exists or do I miss something?

Thanks,
Xingcan

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Question about the custom partitioner

xingcan
Hi Aljoscha,

Thanks for your explanation. I'll try what you suggests.

Best,
Xingcan

On Fri, Jun 16, 2017 at 5:19 PM, Aljoscha Krettek <[hidden email]> wrote:
Hi,

I’m afraid that’s not possible out-of-box with the current APIs. I actually don’t know why the user-facing Partitioner only allows returning one target because the internal StreamPartitioner (which extends ChannelSelector) allows returning multiple target partitions.

You can hack around the API by manually creating your own StreamPartitioner and applying it to the DataStream as DataStream.partitionCustom() and DataStream.setConnectionType() (the first calls the latter) do.

Best,
Aljoscha

On 14. Jun 2017, at 09:09, Xingcan Cui <[hidden email]> wrote:

Hi all,

I want to duplicate records to multiple downstream tasks (not all of them thus the
Broadcasting should not work) in stream environment.
However, it seems that the current custom partitioner can return only one partition index.
Why this restriction exists or do I miss something?

Thanks,
Xingcan


Loading...