This was something I researched extensively already. Once you get down to it, it's a limited batch that can actually run in these systems. Firstly the card needs to have a programmable ROM, then it needs to be able to accommodate the larger Mac ROM, then it actually needs to have the proper flash tools written for it.
Best bet is with the 64mb GeForce2 MX and ATI Radeon product lines, plus a handful of newer ones. All the GeForce2 MX cards i've owned haven't been flashable, unfortunately.

Click to expand...

I have done a few PC to Mac flashes on ATI cards. For some cards you can get special reduced 64K ROMs or whatever the size needs to be to fit. You lose NDRVs which I think were for Mac OS 9 anyway. I used my G5 with an Nvidia AGP card (so as to not confuse with the ATI PCI card), OS X 10.4 and just the standard ATI flashing tools, the ones you use for overclocking I guess.

I no longer have an Nvidia in my G5, now I have a Radeon 9800 Pro, bought off eBay specified "Mac compatible" and probably a flashed card. Works great.