Since I haven't really done much with Quicktime on 68k, the first thing you should do is see what it can handle. If you open an .mov on your 68k, what does Quicktime offer for output formats? If I am not mistaken, Cinepak and animation codecs are maybe old enough to work, but can't be sure. Also don't disregard MPEG1, or MotionJPEG. MotionJPEG is a good tradeoff that you can really dial it down to essentially no compression at all, which results in big files but drastically reduces CPU load. This might be of some help depending on what, if any, acceleration your machine has for video.
If I had to guess I would say you will have more options and success encoding with a PowerPC running Quicktime 4-6. Around QT6 and 7 Apple started to jettison many of the older codecs which would be particularly helpful on an older system. You can probably still get it to work on a modern Mac in QT7, but it might take a bit more effort.