I don't know why it didn't work in Firefox 3.6 and before, but in TenFourFox, font rendering doesn't touch the system libraries TinkerTool would adjust much at all (except for a few calls to the secret CoreText in Tiger to get font properties). Instead, it uses a totally alien font shaper called HarfBuzz, which is out of the operating system's purview. I suspect the previous versions were to do with Cairo (if so, Mozilla 1.7 would have obeyed the setting, maybe 1.8 and 1.8.1 (Firefox 1.5 and 2.0), but definitely not 1.9 (3.0+ and Camino 2.0.x)).