There is a substantial amount of information in the "The Macintosh Quadra 840AV and Macintosh Centris 660AV Computers" Hardware Developer Note: http://developer.apple.com/legacy/mac/library/#documentation/Hardware/hardware2.html. Pages 231 to 289 or about 68 pages of information titled "DSP Operating System".
I would start there. I think that was probably the documentation for developers who wanted to make use of the on-board DSPs. I don't think it will clearly tell you how to use DSPs on NuBus cards, but it might give you some ideas of what to look for.
You'll also need datasheets for the model of DSP with which you are interfacing. And absent developer tools, I imagine most of the work you'd be able to do will be in assembly (or machine code) for the DSP(s) in question.
