Hmm well.

If the memory used by the processes is not what's draining your memory usage... you know that almost all free memory is normally eaten up by UNIX systems for the file system cache, right (beeing freed when a program demands memory)? Maybe you are seeing that?