The AVL invariant does not guarantee that, given any two tree paths, their length differs at most by one unit. They can differ by more than one unit, as shown by the following tree (Fibonacci AVL tree from Wikipedia):
Instead, the AVL invariant only requires that the heights of the two subtrees of any nodes differ at most by one. Note that the height of a (sub)tree is the maximum length of its paths. Therefore, the AVL invariant is equivalent to requiring that, for all nodes, the maximum length of a path in the left subtree and the maximum length of a path in the right subtree differ at most by one unit.
Note how we only take the paths giving the maximum length, and never the paths giving the minimum length.