Extra RAM makes a huge difference, negatively, when it comes to startup time.
This is evident even when going from 1MB/2MB to 4MB. A 2MB Classic with System 6 can boot in under 20 seconds. A 4MB Classic will approach the 30 second mark with the same System 6 installation.
Pluses are notoriously slow with 4MB as well. (Compare to a 1MB Plus for proof).
It's no surprise the SE/30 is like this when accessing that much RAM, even with a faster chip than either of the aforementioned models.
EDIT: Now I sort of want to take my SE/30 beyond 8MB...mmm, must resist the temptation, especially since I need to open it up soon to replace the dead hard drive inside!