There are lots of opinions regarding this, but in my personal experience each new version of the OS has seemed more responsive than the last. I run 10.5 primarily because it runs more software and secondarily because it feels more responsive. In certain things, such as system overhead when running Compressor, 10.5 is definitely better than 10.4, but I suppose most people don't max out all of their CPUs very often. Perhaps the most common way would be via Handbrake, but even the newest version of that doesn't run on 10.4.
My thought is that memory is cheap, so as long as you have plenty of memory, always run the latest OS.
I do still have 10.4 (Server) on a tray-loading iMac G3 with G4 card, but I keep that mostly because it's cool (the motherboard is built into a Tonka truck) and because it was my primary machine for many years. I'd love to be able to run 10.5 on it...