Sorry if someone already pointed out the obvious, but if you are using the original system disk that came with the machine, there may not be a "shut down" command. When I run my floppies that have System 1.0 and System 1.1, there is no shut down command at all. (It freaks me out, frankly. What do you mean I'm supposed to eject the disks first and then turn off the machine WHILE IT IS STILL RUNNING? It's just weird.) I forget if System 2.0 had the Shut Down command. It was an addition to some later version of the system software.
Anyway, if you have a machine which lacks a Shut Down command, and you turn the machine off and on with the disk still in it, it will try to start up from whatever disk is inserted (whether in an external or an internal drive). If your machine has a Shut Down command, it SHOULD eject all disks when you choose Shut Down.
ETA: I do have one external hard drive that, strangely, does the thing where it tries to restart immediately after I choose a menu Shut Down command from a later version of system software. As far as I know, no floppy drive is supposed to do that.