Do linearly dependent features in feature vectors improve the feature vector?

Question

I was reading Wiki on feature vectors, and as far as I can see, it suggests creating new features from already existing features:

Higher-level features can be obtained from already available features and added to the feature vector, for example for the study of diseases the feature 'Age' is useful and is defined as Age = 'Year of birth' - 'Year of death'. This process is referred to as feature construction.

But assuming that you already have included 'Year of birth' and 'Year of Death' as features, will adding 'Age' (that is, 'Year of birth' - 'Year of death') as a feature in the feature vector improve it in any way? I'm thinking not, as the variables are linearly dependent.

If it depends on the machine learning algorithm used, I am mostly interested in SVMs.

Answer 1

I believe it can. Consider the following thought experiment: we are attempting to predict if a person lived to be over 100. Knowing year of birth provides some predictive power (e.g., if they were born after 1912, we know with certainty they did not live to be 100). Year of death also provides some information (people who died closer to the present day are likely to have had a longer lifespan). However "Age" (defined by year of death - year of birth) will be a perfect predictor.

Answer 2

Though edron's thought experiment is nice, it assumes that you do not already have both of those features. If you did, then adding the third feature cannot help, because, as you say, it is linearly dependent. Assume features x1 = Year of birth, x2 = Year of death and x3 = Age = x2-x1. Then any linear predictor gives:

x1*w1 + x2*w2 + x3*w3 = x1*(w1+w3) + x2*(w2-w3)

So, nothing has been gained and we could have learned the same thing with the original features.

A better way to augment the features is to add a nonlinear function of features, such as (x2-x1)^2.

Answer 3

the simple answer/ example is that linear regression alone is considered a basic "learning algorithm" and linear combinations of input variables will have no effect on its performance, ie be redundant. wrt SVM, there are a lot of variants, some basic ones linear. generally with most nontrivial learning algorithms, linear combinations of input vectors are redundant, ie will have no effect on performance of the learning algorithm. also inputs in many cases are typically "conditioned/ smoothed" with a PCA-like step which again would dimensionally reduce any redundant (linear dependent) input vectors.

however, none of this excludes creating nonlinear combinations of input vectors, and yes nonlinear combinations of input vectors can in various cases substantially improve the performance of the learning algorithm. each general area of study/ ML application would tend to have some rules-of-thumb for creating nonlinear combinations. also there are a lot of cases of input variables that are roughly correlated, but not totally, and adding them can improve performance of the algorithm. for example in finance, a lot of variables might roughly "trend with the overall market," but not in a strictly correlated way, and the "noncorrelated parts" may actually have useful information the ML algorithm can harness.