The spacing between the two ROM sockets is different for the Plus and the 128K/512K/512Ke. That's the main reason a single adapter board for all those machines would be problematic.
The sources of the ROM sockets' chip enable and output enable signals are also different between the two types of Mac. The difference doesn't matter when you're doing a normal 128K ROM swap to turn a 512K into a 512Ke, but it does matter when trying to build a sneaky adapter to give you 1 MB of ROM.