More Effective Distributed ML via a Stale Synchronous Parallel Parameter Server

We propose a parameter server system for distributed ML, which follows a Stale Synchronous Parallel (SSP) model of computation that maximizes the time computational workers spend doing useful work on ML algorithms, while still providing correctness guarantees. The parameter server provides an easy-to-use shared interface for read/write access to an ML model's values (parameters and variables), and the SSP model allows distributed workers to read older, stale versions of these values from a local cache, instead of waiting to get them from a central storage. This significantly increases the proportion of time workers spend computing, as opposed to waiting. Furthermore, the SSP model ensures ML algorithm…

• NS
  National Science Foundation

  Awards: 1111142, ISTC-CC
• WD
  Western Digital
• N
  NetApp
• V
  VMware
• ST
  Seagate Technology
• IS
  International Science and Technology Center
• NI
  National Institutes of Health

  Awards: R01GM087694, R01GM
• DA
  Defense Advanced Research Projects Agency

  Awards: FA8750, FA87501220324
• S
  Samsung