That is indeed really confusing. My best interpretation off-the-cuff is that perhaps /HSHK signals are differentiated from "normal" data signals by the line being pulled either up or down for a duration longer than a single byte window? (IE, the IWM raises an error condition if it doesn't get at least X many 0-1 transitions within a certain window, the /HSHK transmission is accomplished by leveraging that? IE, if what the Mac is getting from the HD-20 looks like "data" it is "data", but if the line's held up or down solid for more than x many microseconds, thereby triggering an "error", it's a handshake?)
I don't know if anyone's sat down and compared these fragments of HD-20 documentation to the protocol documentation for SmartPort Apple II hard drives, but might that be worth doing? I know at least one person out there in Internetland makes a replica Smartport drive and I believe they communicate broadly the same way (using an IWM as a UART, etc) with the host as the HD-20, is it really likely they're that much different?(*)
(*Edit: Well, maybe they are. I haven't chased down the manual yet, but at first glance it at least looks to me based on the cable pin details here here that the SmartPort uses the "WRPROT" line as a separate "ACK" signal. I guess I'll put it on my list to download the Apple IIgs firmware reference/poke through the source code and see if the SmartPort handshaking is more straightforward than the phase line state-dance the HD-20 uses. There still might be something useful there. Perhaps they use the same checksum algorithm?)