Question regarding configuring number of network buffers

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Question regarding configuring number of network buffers

Ray Ruvinskiy

The documentation provides the formula #slots-per-TM^2 * #TMs * 4 to determine the number of network buffers we should configure. The documentation also says, “A logical network connection exists for each point-to-point exchange of data over the network, which typically happens at repartitioning- or broadcasting steps (shuffle phase). In those, each parallel task inside the TaskManager has to be able to talk to all other parallel tasks.” Does that say that each parallel task inside the TaskManager talk to all parallel tasks inside the same TaskManager or to all parallel tasks across all task managers? Intuitively, I would assume the latter, but then wouldn’t the formula for determining the number of network buffers be more along the lines of (#slots-per-TM * #TMs)^2?

 

Thanks,

 

Ray

 

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Question regarding configuring number of network buffers

Tzu-Li (Gordon) Tai
Hi!

I’ve looped in Nico (cc’ed) who knows about the configuration of network buffers the best, and could probably explain it to you in more detail.

Cheers,
Gordon


On 7 June 2017 at 5:59:08 PM, Ray Ruvinskiy ([hidden email]) wrote:

The documentation provides the formula #slots-per-TM^2 * #TMs * 4 to determine the number of network buffers we should configure. The documentation also says, “A logical network connection exists for each point-to-point exchange of data over the network, which typically happens at repartitioning- or broadcasting steps (shuffle phase). In those, each parallel task inside the TaskManager has to be able to talk to all other parallel tasks.” Does that say that each parallel task inside the TaskManager talk to all parallel tasks inside the same TaskManager or to all parallel tasks across all task managers? Intuitively, I would assume the latter, but then wouldn’t the formula for determining the number of network buffers be more along the lines of (#slots-per-TM * #TMs)^2?

 

Thanks,

 

Ray

 

Loading...