Sorted! Downloaded 'Serial Switch' control panel from Apple's site and set the mode to compatible. Mac286 software then loaded without crashing but all I got was a spinning hour glass and menu options were disabled. Couldn't find any other software options to change so I thought I'd pull the cards to see if there were any jumpers to change. Nothing obvious but I did notice a nice shiny gold maths co-processor....8087....Hmm...286 CPU! Pulled the 8087 and tried again. Success!

So, if anybody else is trying to run an AST Mac286 on a IIfx, you need to be running in 24bit addressing, Serial ports set to compatible and make sure nobody has stuffed the wrong co-processor in!