Parameter Server
A programming framework designed for distributed machine learning that shards model parameters across multiple server nodes; training nodes read and write parameters in parallel through pull and push operations, supporting training of very large models.
Timeline (last 90 days)
Parameter Server is a programming framework designed for distributed machine learning, first proposed by researchers at Carnegie Mellon University.
The core idea of the parameter server is to partition model parameters into multiple shards distributed across a set of server nodes, with training nodes interacting with the parameter server through pull and push operations.
The drawbacks of the parameter server architecture are: when model parameters reach the hundred-billion level, the communication volume between workers and parameter servers is enormous, making network bandwidth a bottleneck; popular parameters tend to concentrate on a few nodes to form hotspots, causing load imbalance; and it is usually tightly coupled with training frameworks, making it difficult to perform low-level optimization for specific access patterns.
Stream processing frameworks (such as Flink) and machine learning training frameworks (such as parameter servers) widely adopt checkpoint mechanisms.
All Facts (4)
Parameter Server is a programming framework designed for distributed machine learning, first proposed by researchers at Carnegie Mellon University.
50%UnverifiedThe core idea of the parameter server is to partition model parameters into multiple shards distributed across a set of server nodes, with training nodes interacting with the parameter server through pull and push operations.
50%UnverifiedThe drawbacks of the parameter server architecture are: when model parameters reach the hundred-billion level, the communication volume between workers and parameter servers is enormous, making network bandwidth a bottleneck; popular parameters tend to concentrate on a few nodes to form hotspots, causing load imbalance; and it is usually tightly coupled with training frameworks, making it difficult to perform low-level optimization for specific access patterns.
50%UnverifiedStream processing frameworks (such as Flink) and machine learning training frameworks (such as parameter servers) widely adopt checkpoint mechanisms.
50%