Been there, done that. I even tried the $ symbol in front of the hex password. Only thing I did not try was a $ for every byte. 

my password for 64bit is DEADBEEFBA. simple. I tried $DEADBEEFBA and it didnt work. and DEADBEEFBA as itself doesnt work. I tried 128bit as well, using a longer hex key of course. Same results. 

I treat all WEP as HEX. as the password generator tool in the Linksys router is going to be different from other things.