Suppose that we take a neural network of a given topology, and run it through two training processes, obtaining two different sets of converged weights at the end of the training.
What is a good way to measure the difference between the two sets of weights?
If this were a curve-fit where the parameters were largely orthogonal, we could simply treat the parameters as a cartesian space and compute the length of the vector difference between the sets.
However, the weights in a neural network are often not at all orthogonal. There are trivial redundancies, in that we can reorder nodes within a layer, or multiply the input weights to a given node by a constant factor and the output weights by its inverse. Even if we remove these by putting the weights in some canonical form (normalizing and sorting them, for instance), I can't see any reason to assume that we won't have plenty of remaining non-orthogonality. As a result, two sets of neural-network weights that have a large vector-difference length may be essentially equivalent in practice.
An alternate conceptual approach would be to numerically integrate the difference between the output values over the entire input space, but for typical neural networks this space has high enough dimension for such an integration to be intractable. So we'd need to do some sort of sampling-based approach, which introduces more handwaving about whether it's a meaningful measure.
Is there any existing consensus on what's a good measure? Or, for that matter, any existing practice for what other researchers have used?