The Apple II is *much* slower, narrower, simpler, and better documented than the SE/30, and has a quite straightforward memory map. The SE and Plus are also fairly simple machines (Somewhat arcane memory management aside, of course. The 68000 CPU actually has some specific support built into its bus to make it easily backwards-compatible with slow 8-bit peripheral chips like the VIA. That sort of thing takes glue on the 68030.) So I dunno, personally I could see some value in using something like that as a testbed for making a FPGA act like a peripheral. (In the case of the Apple II, hey, make the first ever 512MB DDR "Slinky" RAM card!) But hey, whatever works.
I would strongly recommend looking at the work the MESS developers have done in documenting the Mac hardware:
http://mess.redump.net/mess:driver_info:mac_technical_notes
(There are many useful links off that page as well, and there's probably more useful information in the source code of the emulator itself.) and at the old Mac II-era hardware tech notes, like the IIsi, IIci, Classic II, etc. They don't go into that much detail but several of them do at least have some helpful diagrams of how the physical address space is mapped into the 24 and 32 bit logical address spaces depending on how the machine is configured. It's all done with the MMU. (Although, it's also worth remembering that in addition to the PMMU there's a hardware page-flipping device that on initial startup maps a section of ROM to 0x00000000 for IPL. You can probably ignore that for the most part but when you're reading about the boot sequence it's probably good to remember it exists so you don't get confused by it.) At boot-time mapping tables are compiled and loaded into it the PMMU... I have the IIsi DevNote open now and it shows how the PMMU is even used to paper over the physical gap between the first and second RAM banks. (This mapping happens even in 24 bit addressing mode.) Unless the "Dirty ROM"-based Macs behaved fundamentally differently, which I don't see a whole lot of evidence for, then I'd assume the SE/30's ROM does similar things with the PMMU. (Or the Apple-proprietary "HMMU" in the original Mac II's case.)
There is a "RAM BASE" global variable, but I'm skeptical that just changing it to point to a bank of memory "somewhere high" would do what you want. (The one reference I saw for it said something about the variable being used by the Trap Dispatcher, which is possibly the scariest and most obscure piece of code in Mac OS.) MacOS is designed around using fixed places in low memory for global variables, and while I suppose it's possible you might be able to use that RAM-BASE global to open a "hole" above the low memory area for some purpose (drivers?) I'd be really surprised if you could set it and move the whole OS to some arbitrary place not mapped by the MMU into what MacOS considers "RAM space".
I'm pretty sure what "Mode32" mostly does is fix some bad/missing MMU maps so the early machines operate properly in 32 bit mode. If that's the case then it may well be possible to make a custom Mode32 that can operate as an OS patch with stock SE/30 ROMs, although it may almost be simpler to make a custom ROM upgrade based on the IIsi/ci 32-bit clean ROMs that natively supports discovering and mapping "high" memory.
Or I could be *completely* wrong about everything. Your mileage may vary.