Continue batch job with streaming job

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Continue batch job with streaming job

Sergei Poganshev
Is there a way to make a checkpoint/savepoint after the batch job has finished and then run the job in a streaming mode with state that has been initialized in batch mode?

Or more generally speaking, what are the battle-tested solutions to "job initialization" problem, especially when there are terabytes of data that have to be processed before the job can be turned into streaming mode? Do any examples exist?
Reply | Threaded
Open this post in threaded view
|

Re: Continue batch job with streaming job

Andrey Zagrebin
Hi Sergei,

Batch mode does not support state.
At the moment there is no “out of the box” way to bootstrap the state for streaming job.
You can have a look at Bravo tool [1] which is developed to address this problem.

Best,
Andrey

[1] https://github.com/king/bravo

> On 29 Oct 2018, at 22:29, Sergei Poganshev <[hidden email]> wrote:
>
> Is there a way to make a checkpoint/savepoint after the batch job has finished and then run the job in a streaming mode with state that has been initialized in batch mode?
>
> Or more generally speaking, what are the battle-tested solutions to "job initialization" problem, especially when there are terabytes of data that have to be processed before the job can be turned into streaming mode? Do any examples exist?