The question is important to understanding the data to use in the observation, and also in understanding which data to remove from consideration.

For example in the wikipedia editing, the time matrix clearly reflects times when few edits take place, between midnight and about 5AM. Experience says that this is sleeping hours. Note that the spikes occur typically after 8PM. Wikipedia editing is not something that most folks do, so why 8PM? Well as a nerd, I find that people leave me alone then and I can do things that require more concentration. So what are the social implications of this?

I love the attention to outliers as the source of more information. I teach people that in my classes dealing with production issues for semiconductors. Outliers give you more information about any process.

Parsing the data can reveal information, but it is the interpretation of the information that often gets us in trouble. Ignoring the raw data's full spectrum may obscure the root cause of the peaks and valleys although the important piece may not reflect any peaks and valleys at all, but the sum of the artifacts produces the "rogue wave" in the data you are examining.

Thanks for your comment. I am looking forward to your next posts.

Igor,

Thank you for the links that you provided, they are valuable. It will take some time to digest all the material.

I actually had the matrix factorization site bookmarked on my toolbar (haven't had a chance to explore it yet). Thanks for the info and resources.

-Nelson

We might release it in the future once it gets tidied up a bit. In the meantime, you can find some MATLAB code at http://tfocs.stanford.edu/demos/rpca/

The algorithm is different from the "pcaPP" package in R (theirs is based on work by Croux and Ruiz-Gazen, 2005). We implemented a very simple iterative thresholding method, but you can try out some fast accelerated proximal gradient techniques. See http://yima.csl.uiuc.edu/psfile/rpca_algorithms.pdf

-Nelson