Multiple Taskmanagers per node for standalone cluster

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view

Multiple Taskmanagers per node for standalone cluster

Ethan Li

Does/did anyone try to set up a standalone cluster with multiple TaskManagers per node? We are working on moving to flink-on-yarn solution. But before that happens,  I am thinking about the following setup to  get jobs isolated from each other

1) multiple taskmanager per host
2) 1 taskSlot per TaskManager

Currently we have 1 TaskManger per node and many taskSlot per TM, tasks from different job will be scheduled into one JVM process and it’s basically impossible to debug. One bad job will kill the whole cluster.

Could you please share if you have any experience on this and what’re the problems that might have?

Thank you very much. Really appreciate it.