Sounds like a bit of problem determination is required.

Try producing a list of boot up times with varying amounts of memory.

Also, duplicate that same list without virtual memory on.