The delay is normal, due to the startup memory test - longer for more RAM. You can disable this test (I think you hold down option + command while double clicking on the memory control panel).

Have you tried running only a pair of the 32MB sticks and seeing if any errors at startup result? Money is on the "long" Memory test will show an error.

JB