Restoring Queryable State

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Restoring Queryable State

Philip Doctor

Hello,

My job differs slightly from example Queryable State jobs.  I have a keyed stream and I will emit managed ValueState at certain points at runtime but the names aren’t entirely known beforehand. 

 

I have check pointing enabled and when I restore from a check point, everything *almost* works.  In my code I took a known ValueState name, hard coded a new ValueStateDescriptor, get state from the RuntimeContext, and I’ll see that my value is correctly restored.  However, that value is not queryable unless I create a new ValueStateDescriptor that is queryable.  So if I knew all of the ValueState names ahead of time, I could simply build a ValueStateDescriptor for each emitted value state name, and set each to queryable and presumably life would be good.  But returning to the problem, I don’t know all the value state names.

 

Is there a way that, on restore, the keyed managed ValueStates that I am restoring can also be queryable as they were prior to the crash?

 

Thanks.

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Restoring Queryable State

Tzu-Li (Gordon) Tai
Hi Philip,

I don’t think that is possible right now. The main thing is that Flink currently doesn't store information about whether or not a registered state is queryable or not. So, it wouldn’t be queryable until a new StateDescriptor is provided for that state.

Would you be able to know the state names after the checkpoint was taken? Currently, as you mentioned, the only possible workaround is to register them in `initializeState` in the restored job. The downside of this would then be that you would need to have different code for the restored job.

Cheers,
Gordon


On 1 June 2017 at 11:41:39 PM, Philip Doctor ([hidden email]) wrote:

Hello,

My job differs slightly from example Queryable State jobs.  I have a keyed stream and I will emit managed ValueState at certain points at runtime but the names aren’t entirely known beforehand. 

 

I have check pointing enabled and when I restore from a check point, everything *almost* works.  In my code I took a known ValueState name, hard coded a new ValueStateDescriptor, get state from the RuntimeContext, and I’ll see that my value is correctly restored.  However, that value is not queryable unless I create a new ValueStateDescriptor that is queryable.  So if I knew all of the ValueState names ahead of time, I could simply build a ValueStateDescriptor for each emitted value state name, and set each to queryable and presumably life would be good.  But returning to the problem, I don’t know all the value state names.

 

Is there a way that, on restore, the keyed managed ValueStates that I am restoring can also be queryable as they were prior to the crash?

 

Thanks.

Loading...