The components in the SE's PSU are placed very close together, most of the capacitors touching. I would think this makes it more difficult for air to travel through the unit. Another possible problem is the close proximity of the PSU to the CRT, which could potentially block some of the airflow. This is in comparison to the exposed power supply in earlier models where these problems didn't really exist. So without a fan I would be concerned about damaging components in the PSU due to excessive trapped heat.

It might also be important to note that most floppy drives do heat up during operation (although probably more so 5.25" than 3.5" drives). Apple included a fan with the SE to try and avoid any reoccurrence of the heat problems users experienced with previous compact models, and also because of the addition of a built-in hard drive.