You're missing the point of Dylan. YES his voice is unattractive, but he isn't using it in the same way most other vocalists do. That's what makes him so interesting - he's a poet who happens to write songs. He's effectively singing poetry, which gives it a whole new dimension. The tone of words when written and when sung are so different and Dylan takes this to an different level. He uses all the conventions most modern poets do, but as they're sung and backed up by music it makes it much more interesting. It opens up for more ambiguous readings to his lyrics as well. Like 'Stuck inside of mobile with the memphis blues again', everybody expects him to hold the note when he sings 'again' in the chorus (Like almost any other vocalist would) but because he cuts it short, it opens up other meanings (Which are supposed to be figured out on a personal basis, as all great art can be - which means I am not going to 'tell you' what they are because I know as much as you). The lyrics are the primary objective in his work, they are not just words to fill songs. Sometimes he seemingly sacrifices the lyrical reading to fit in with the music, like in 'Tangled up in blue' and 'It's alright ma, I'm only bleeding' and a few others, but that great ambiguity can still be found in those too. On top of this of course, like you said, it doesn't hurt that the songs are incredible as well. If you meant that the SONGS are horrible on the ears, that's bollocks!
The thing that differs him from almost all rock artists is that he is primarily a poet acting as a rock star, rather than a rock star pretending to be a poet. Most rock artists just need something to sing.
All MO as well, of course.