[SURVEY] How do you use ExternallyInducedSource or WithMasterCheckpointHook

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view

[SURVEY] How do you use ExternallyInducedSource or WithMasterCheckpointHook

Biao Liu
Hi everyone,

I would like to reach out to the user who uses or is interested in `ExternallyInducedSource` or `WithMasterCheckpointHook` interfaces.

The background of this survey is I'm doing some reworking of `CheckpointCoordinator`. I encountered a problem that the semantics of `MasterTriggerRestoreHook#triggerCheckpoint` [1] might be a bit confusing. It looks like an asynchronous invocation (value returned is a future). But based on the description of java doc, the implementation could be synchronous (maybe blocking) or asynchronous. It's OK for now. However it makes the optimization more complicated, to take care of synchronous and asynchronous invocation at the same time [2].

I want to make the semantics clearer. Here are some options.
1. Keeps this method as the current. But emphasize in java doc and release note that it should be a non-blocking operation, any heavy IO operation should be executed asynchronously through the given executor. Otherwise there might be a performance issue. In this way, it keeps the compatibility.
2. Changes the signature of this method. There will be no executor and completable future in this method. It could be blocking for a while. We will execute it in an executor outside. This also makes things easier, however it breaks the compatibility.

At this moment, personally I intend to choose option 1.
If you depends on these interfaces, please let me know your opinion. Any feedback is welcome!

Biao /'bɪ.aʊ/