Not too surprising there. IBM MDA/Hercules and the compact Mac are both roughly 350 line video standards but MDA runs at 50hz while the Mac is a hair over 60hz vertical refresh, with a corresponding increase in hsync. So roughly speaking you're "overclocking" an MDA monitor by about 20% if you connect it to a Mac motherboard. I'm sure some monitors stood up to that better than others.
As for adapting the Compact Mac's output for more modern monitors, probably the most straightforward approach is a simple scan doubler. Here's the schematic for one that's far more complicated than you'd need for this because it includes an NTSC decoder and handles color. Basically cut out the entire NTSC decoder, axe the A/D converters as well, use only one line buffer (instead of 3), and dumb down your D/A output converter to a simple resistor. (It would probably be pretty trivial to replicate this all onto an FPGA; since it's single-bit monochrome your line memory only needs to hold about 64 *bytes*. Of course, on the input side you'll need to adapt it to the Compact Mac's oddball hsync output, per a circuit like Pina's.) This would give you a double-scanned (electrically) VGA compatible signal at a (roughly) 43khz hsync. The result *will* be an oddball video mode and LCD panels which are strict about only supporting VESA-standard video modes won't like it but it's well within the capability range of more forgiving multisyncs. The HSYNC would be about the same as 640x480@85hz, or a little higher than 800x600@60hz and well under 1024x768@60hz.