Modern neural networks tend to use large numbers of coefficients and neurons, which increases processing power requirements. Models with hundreds of thousands of parameters are common. Building a neural network structure is largely a manual process that involves tuning many variables at once to find a good balance between model size and accuracy. These variables include:
- Random seed
- Number of neurons
- Number of layers
- Activation function (Sigmoid, ReLU, and others)
- Learning rate
- Number of epoches
- Cross validation folds
- Dropout
Most modern neural networks use a fixed architecture defined by the researcher and rely on stochastic gradient descent (with minor variations) to optimize neuron parameters. The architecture itself does not change during training.
Because only the parameters are optimized while the structure stays fixed, networks often end up larger than necessary. This increases prediction costs due to redundant computations within the network. Two main approaches to reducing network volume that can currently be distinguished are:
- Optimizing the structures of already-trained networks (pruning) — Methods that follow this approach typically work by removing neurons and connections from an already-trained network based on certain criteria. This reduces the network size but comes at the cost of accuracy. The full-size network still has to be trained first, so the size reduction only happens after training, not during it.
-
Automated neural architecture search (NAS) — This approach generates optimized network architectures that can match or exceed the performance of manually designed architectures. In practice, most NAS methods work by evaluating a set of candidate architectures, training a full model for each one, and selecting the best result. The typical process is as follows:
This process is resource-intensive, so in practice the search space of possible architectures is heavily restricted. The result is a coarse search that often produces a less-than-optimal architecture. Cross-validation, which is needed for consistent results, multiplies this overhead further.
Another obstacle to obtaining an efficiently sized, highly accurate model is the choice of the optimization algorithm. The widely-known problem of local extremes and plateaus significantly reduces the efficiency of using stochastic gradient descent for these purposes. On top of that, results are sensitive to hyperparameters such as learning rate, batch size, and weight initialization. Determining when training is complete is also not straightforward. Together, these factors add uncertainty to each step and increase the overall cost of the process.