It does indeed run cooler. But think about it; the old 8MHz Macs with no fans had hella ventilation, and they still suffer the ill effects of overheating. Your SE has none of those vents, so with no fan, it would suffer worse. I'd just get a quieter fan if it bothers you that much; don't run it for any real length of time with no fan.

In general, if it has a fan in/on it, it's prolly meant to have it there for some reason or another.