Making broadcast state queryable?

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Making broadcast state queryable?

Oytun Tez
Hi there,

Can we set a broadcast state as queryable? I've looked around, not much to find about it. I am receiving UnknownKvStateLocation when I try to query with the descriptor/state name I give to the broadcast state.

If it doesn't work, what could be the alternative? My mind goes around ctx.getBroadcastState and making it queryable in the operator level (I'd rather not).

---
Oytun Tez

M O T A W O R D
The World's Fastest Human Translation Platform.
Reply | Threaded
Open this post in threaded view
|

Re: Making broadcast state queryable?

Yu Li
Hi Oytun,

Sorry but TBH such support will probably not be added in the foreseeable future due to lack of committer bandwidth (not only support queryable broadcast state but all about QueryableState module) as pointed out in other threads [1] [2].

However, I think you could open a JIRA for this so when things changed it could be taken care of. Thanks.


Best Regards,
Yu


On Tue, 13 Aug 2019 at 20:34, Oytun Tez <[hidden email]> wrote:
Hi there,

Can we set a broadcast state as queryable? I've looked around, not much to find about it. I am receiving UnknownKvStateLocation when I try to query with the descriptor/state name I give to the broadcast state.

If it doesn't work, what could be the alternative? My mind goes around ctx.getBroadcastState and making it queryable in the operator level (I'd rather not).

---
Oytun Tez

M O T A W O R D
The World's Fastest Human Translation Platform.
Reply | Threaded
Open this post in threaded view
|

Re: Making broadcast state queryable?

Oytun Tez
Thank you for the honest response, Yu!

There is so much that comes to mind when we look at Flink as a "application framework" (my talk in Flink Forward in Berlin will be about this). QS is one of them (need-wise, not QS itself necessarily). I opened the design doc Vino Yang created, I'll try to contribute to that.

Meanwhile, for the sake of opening our state to outside, we will put a stupid simple operator in between to keep a duplicate of the state...

Thanks again!





---
Oytun Tez

M O T A W O R D
The World's Fastest Human Translation Platform.


On Tue, Aug 13, 2019 at 6:29 PM Yu Li <[hidden email]> wrote:
Hi Oytun,

Sorry but TBH such support will probably not be added in the foreseeable future due to lack of committer bandwidth (not only support queryable broadcast state but all about QueryableState module) as pointed out in other threads [1] [2].

However, I think you could open a JIRA for this so when things changed it could be taken care of. Thanks.


Best Regards,
Yu


On Tue, 13 Aug 2019 at 20:34, Oytun Tez <[hidden email]> wrote:
Hi there,

Can we set a broadcast state as queryable? I've looked around, not much to find about it. I am receiving UnknownKvStateLocation when I try to query with the descriptor/state name I give to the broadcast state.

If it doesn't work, what could be the alternative? My mind goes around ctx.getBroadcastState and making it queryable in the operator level (I'd rather not).

---
Oytun Tez

M O T A W O R D
The World's Fastest Human Translation Platform.
Reply | Threaded
Open this post in threaded view
|

Re: Making broadcast state queryable?

Yu Li
Good to know your thoughts and the coming talk in Flink Forward Berlin Oytun, interesting topic and great job! And it's great to hear the voice from application perspective.

Could you share, if possible, the reason why you need to open the state to outside instead of writing the result to sink and directly query there? In another thread there's a case that sink writes to different kafka topics so state is the only place to get a global view, is this the same case you're facing? Or some different requirements? I believe more attention will be drawn to QS if more and more user requirements emerge (smile).

Thanks.

Best Regards,
Yu


On Wed, 14 Aug 2019 at 00:50, Oytun Tez <[hidden email]> wrote:
Thank you for the honest response, Yu!

There is so much that comes to mind when we look at Flink as a "application framework" (my talk in Flink Forward in Berlin will be about this). QS is one of them (need-wise, not QS itself necessarily). I opened the design doc Vino Yang created, I'll try to contribute to that.

Meanwhile, for the sake of opening our state to outside, we will put a stupid simple operator in between to keep a duplicate of the state...

Thanks again!





---
Oytun Tez

M O T A W O R D
The World's Fastest Human Translation Platform.


On Tue, Aug 13, 2019 at 6:29 PM Yu Li <[hidden email]> wrote:
Hi Oytun,

Sorry but TBH such support will probably not be added in the foreseeable future due to lack of committer bandwidth (not only support queryable broadcast state but all about QueryableState module) as pointed out in other threads [1] [2].

However, I think you could open a JIRA for this so when things changed it could be taken care of. Thanks.


Best Regards,
Yu


On Tue, 13 Aug 2019 at 20:34, Oytun Tez <[hidden email]> wrote:
Hi there,

Can we set a broadcast state as queryable? I've looked around, not much to find about it. I am receiving UnknownKvStateLocation when I try to query with the descriptor/state name I give to the broadcast state.

If it doesn't work, what could be the alternative? My mind goes around ctx.getBroadcastState and making it queryable in the operator level (I'd rather not).

---
Oytun Tez

M O T A W O R D
The World's Fastest Human Translation Platform.