(Contributed by George S.)
Collinearity of the predictors variables can also cause Ill-conditioned matrices. In other words, if one predictor is linearly related to another predictor, then you will have a ill-conditioned matrix
From StatSoft's online textbook (http://www.statsoft.com/textbook/multiple-regression/#amulticollinearity)
Imagine that you have two predictors (X variables): (1) weight in pounds and (2) weight in ounces. Obviously, our two predictors are completely redundant; weight is one and the same variable, regardless of whether it is measured in pounds or ounces. If you were to attempt to regress these two variables to a persons height, then you would have an ill-conditioned matrix since one weight variable (say pounds) is linearly related to the other weight variable (ounces).
Three ways to check multicollinearity:
check pairwise correlations between all the predictor variables
compute the Variance Inflation Factor statistic
mean center all predictor variables.
See page 16 here for more info
See http://www.statisticalhorizons.com/multicollinearity for an interesting article of when multicollinearity may not be an issue.