Profiling Neural Networks to improve model training and inference speed

Part 3: Quantifying the improvements in your model

In this series, we have been exploring how to debug and improve the performance of Convolutional Neural Networks (CNNs). In part 1, we studied how to profile the performance of a neural network using the TensorBoard profiler. In part 2, we examined how to follow up on one of the recommendations from TensorBoard profiler: turning on mixed precision training for our model. In this part we aim to discuss how we can scientifically and methodically quantify the improvements that a change to a CNN model can deliver, and how to conclude whether the same change is desirable or not.

Quantifying model changes

Here are a few principles on how to scientifically quantity model changes:

1. Make an Apples to apples comparison: Make sure that you thoroughly know and understand what you are comparing. Often there are subtle changes that happen in the background which can cloud any comparisons that we make between two experiments. In the context of this work, it is important to check that the training is performed on the same type of GPU, on the same training dataset, and for the same number of epochs.

2. Be careful about run to run variations: When we run two experiments, the end result is rarely the same. The differences occur due to many kinds of random and stochastic processes that always happen on a system. Hence, it is important to not judge the results for an experiment from a single run of the experiment. Rather, one should run the experiment multiple times. Results presented from multiple experiments give the readers a much higher degree of confidence.

3. Use multiple data points to compare and contrast results. Once we have the results from multiple runs of an experiment, we need to use different measures to interpret the results. Often, the mean of the results is used as the sole point of comparison. However, usually mean is not enough. Additional metrics that may be useful for comparison are: i) Standard Deviation, ii) Median, iii) Percentiles: such as 90th percentile.