Well part of the problem is the enclosed PSU. So the extra airflow is really needed, IMO, to help cool the enclosed PSU.
This is an interesting quandary.
First I would say to wgoodf that the second floppy disk would contribute to similar airflow problems as the hard disk, though without generating the additional heat. If I were to even consider running an SE without a fan, I would most likely remove everything I could from the interior, including both floppy drives and run the SE with an external floppy and/or disk drive, which sort of defeats the purpose.
Second, the Apple III was often claimed to have failed due to heat related issues caused by the PSU enclosed Aluminum chassis. However, Manock proved there was adequate heat dissipation and the primary culprit turned out to be a major logic board "fineline" technology design problem. Interestingly enough, a similar solution was implemented in the Macintosh: a heat-sink was installed along the top of the original analogue boards to achieve the same effect, but was quickly removed as it had the opposite effect of blocking the ventilation on top of the Mac. Theoretically, the metal cased PSU in the SE might actually work more akin to the Apple III's chassis, conveying the PSU's heat to its case to better cool it. The case is well perforated on all sides (though the SE chassis actually obscures the bottom perfs somewhat), and the side panel could be removed to further enhance air flow, assuming it does not better dissipate heat in place. Interestingly enough, the SE's PSU is very similar to that used in the HD20, except that the HD20's is fully enclosed and sits beneath the circuit board and the air intake vents. So, when the fan is in operation, it pulls air across the top of the circuit board only, but does nothing to pull air up from around the base of the PSU, suggesting the case itself is acting as a heat heat sink.
While this is all interesting speculation and conjecture, the SE was designed to operate with a fan, and given the quieter alternatives available (even at the time), it's best to preserve the operational life of this 20+ year old Mac by using it as intended.